Parrot Count :M

[a cut of domestic sheep prime]

SUCCESS:
In this thread there are:
86 parrots with hats
156 normal parrots
242 total parrots

Seriously, this is typical Windows user criticism .

It does indeed look pretty. A pity, though, that it “diminishes the need to access the terminal” as simple python scripts are designed for the terminal. That's why it has one nonetheless. Terminal is your friend.

Terminal is my friend in the way that Laci Peterson's killer was her husband. (parrot #243)

On Windows, you double-click to install one thing. On Linux, you type one line to install everything you will ever need, with updates for all eternity.

That's where I was running into the hassle though because apparently you need to install certain modules via the terminal and it wasn't working for me, but I digress. I may play around with it some more later.

Thanks loads for the help.

If not for you, I'd have only had imweasel's oh-so-helpful advice to go on.



Update:

My research is complete. I analyzed 3 different Codex CYOA Let's Play mega threads and found a startling trend...

As you can see by the chart, the ratio of parrots to pages has shifted substantially with each successive Let's Play - Epic being the oldest and Overlord being the newest.

The implications are clear: I have way too much time on my hands today. (parrot #244)

 

[Nevill]

My research is complete. I analyzed 3 different Codex CYOA Let's Play mega threads and found a startling trend...

Test Spiral. That was the turning point. We must analyze where we went horibly wrong.

Edit: 600 parrots in Overlord!? We are doomed.

 

[imweasel]

If not for you, I'd have only had imweasel's oh-so-helpful advice to go on.

I said use Python and what are using now? Oh, a Python script, which somebody wrote for you.

It can be done on Windows too you damn moron and I was just pointing that out because you were whining about being a Linux noob. Not like I'm going to waste my time to write a Windows friendly script for you or anything.

Saying that in 2015. Moar threads -> faster -> better. Multiprocessing uses processes, not threads, by the way.

One thread uses one process. You only really need one thread resp. one process for this simple task. Duh.

 

[Cowboy Moment]

I said use Python and what are using now? Oh, a Python script, which somebody wrote for you.

It can be done on Windows too you damn moron and I was just pointing that out because you were whining about being a Linux noob. Not like I'm going to waste my time to write a Windows friendly script for you or anything.


One thread uses one process. You only really need one thread resp. one process for this simple task. Duh.

Tasks bound by I/O response time rather than throughput, like http requests to websites, generally benefit from parallelization for obvious reasons. This case would actually be faster with threads instead of processes, as it's not affected by the GIL. You could also remove all the synchronization code by using a package like futures instead of multiprocessing, but whatever, if it works then it works.

 

[Melcar]

Once again, serious motherfucking research for a serious motherfucking subject, all made possible by the motherfucking Codex.

 

[a cut of domestic sheep prime]

Indeed. (parrot #245)

I said use Python and what are using now? Oh, a Python script, which somebody wrote for you.

That's like a surgeon telling a sick person that the operation can be easily done with a scalpel and then walking out of the room. It's a dickhead, know-it-all move that helps no one.

Guys like you are in tech forums all over the internet. You use people's questions as an opportunity to feel superior for ten seconds and then flit off like a faggot.

It can be done on Windows too you damn moron and I was just pointing that out because you were whining about being a Linux noob. Not like I'm going to waste my time to write a Windows friendly script for you or anything.

No shit. Except it wasn't working in windows. Nevill had done the research and said that without Python knowledge, it might be better to just install linux. And I figured that since the guy who wrote the code ran it on linux and it worked fine, then maybe I should use linux and see if it runs. Excuse me for not knowing how to code in a language that is useless for my work/life - except in this one obscure situation for fun.

 

[a cut of domestic sheep prime]

Test Spiral.

85 parrots with hats
697 normal parrots
782 parrots

I've reordered the oh-so-important research by time:

According to my calculations, after a few Let's Plays, the thread will consist of ALL PARROTS and no pages. (parrot #246)

 

[imweasel]

That's like a surgeon telling a sick person that the operation can be easily done with a scalpel and then walking out of the room. It's a dickhead, know-it-all move that helps no one.

Wut.

Are you pissed because because I didn't waste my time to create a script for you because you are too fucking stupid or lazy to write one yourself? I pointed you in the right direction: I told you to use Python, now you are using Python... but strangely enough only after you went on about some retarded bullshit that you won't use Python for really no reason at all. That said, fuck off you dumbfuck lazy ass faggot.

Tasks bound by I/O response time rather than throughput, like http requests to websites, generally benefit from parallelization for obvious reasons. This case would actually be faster with threads instead of processes, as it's not affected by the GIL. You could also remove all the synchronization code by using a package like futures instead of multiprocessing, but whatever, if it works then it works.

You're right, but I don't think it is worth the bother if you only want to occasionally count parrots in random threads on the Codex. But opinions.

 

[a cut of domestic sheep prime]

Wut.

Are you pissed because because I didn't waste my time to create a script for you because you are too fucking stupid or lazy to write one yourself? I pointed you in the right direction: I told you to use Python, now you are using Python... but strangely enough only after you went on about some retarded bullshit that you won't use Python for really no reason at all. That said, fuck off you dumbfuck lazy ass faggot.

Dude, there are any number of languages that could do this. The fact is that I don't know them. I'm sure you could do the same thing if you programmed your own webspider, but unless you actually know how to do it, the information is useless.

"Hey, you can make a million dollars if you program your own stock market prediction algorithms, but I'm not going to waste my time telling you how so that information is worthless. Bye. Herpderp."

If you don't have anything helpful to say, maybe you should shut the fuck up?

And it's not laziness - I fucking installed Linux when I have no experience with it and no use for it - it's the fact that I didn't know Python. If I did, I wouldn't have had to ask for help. Duh.

You're right, but I don't think it is worth the bother if you only want to occasionally count parrots in random threads on the Codex. But opinions.

Only on the codex can you have a slap fight over parrot counts in python.

 

[Melcar]

are serious business.

 

[a cut of domestic sheep prime]

Nobody understands my love of parrots. I may be parrotkin.

edit: oh, wow, that's apparently a thing.

 

[imweasel]

"Hey, you can make a million dollars if you program your own stock market prediction algorithms, but I'm not going to waste my time telling you how so that information is worthless. Bye. Herpderp."

If you don't have anything helpful to say, maybe you should shut the fuck up?

So you are going to keep on bitching at me because I told you to use Python (which you now are) but didn't waste my time to write a script for your retarded lazy ass?

God you are such a mentally retarded dumbfuck.

 

[a cut of domestic sheep prime]

FFS. Thanks again for your helpful advice, captain obvious. You seriously belong locked up in a Linux help forum were all your posts will consist of "can easily be done with the terminal. ".

 

[imweasel]

No, I will join the Linux help forum and post parrot swastikas all day, so I don't have to hear retards like lambchop bitch at me because I actually mentioned something in my post that was on topic.

 

[a cut of domestic sheep prime]

 

[Melcar]

I miss my parrots.

 

[a cut of domestic sheep prime]

i ate your parrots with chianti and fava beans

 

[kaizoku]

Easy for unix master race:

remove-trailing-slashes() {
  local ret="${1%[!/]*}"
  echo "${ret}${1:${#ret}:1}"
}
threadUrl="$(remove-trailing-slashes "$1")"
parrotRegex='img src="/forums/smiles/('\
'bird\.gif|'\
'wally_the_prestigious_monocled_bird\.gif'\
')"'

pageNumber=1
totalCount=0

{
for i in $(seq 100); do
  pageUrl="$threadUrl"/
  if [ $i -gt 1 ]; then
    pageUrl="$threadUrl"/page-$i
  fi

  parrotCount="$(curl -i "$pageUrl" 2>/dev/null |
      tee >(sed '2,$d' >3) | grep -E -o "$parrotRegex" | wc -l)"
  if grep 307 <3 >/dev/null; then
    break
  fi
  totalCount=$(($totalCount + $parrotCount))
done
}<3

echo $totalCount

With parallelism!

import os, urllib.request, sys, re, collections, multiprocessing as mp

ParrotType = collections.namedtuple("ParrotType", ["groupName", "description"])
Result = collections.namedtuple("Result", ["pageNumber", "counts"])
ProcessPoolSize = 5

ParrotRegex = re.compile(r'img src="/forums/smiles/('
r'(?P<normal>bird\.gif)|'
r'(?P<hat>wally_the_prestigious_monocled_bird\.gif)'
r')"')
Types = [ParrotType("hat", "Parrot with hat"),
    ParrotType("normal", "Normal parrot")]


def countInString(ParrotRegex, Types, counts, string):
  for match in ParrotRegex.finditer(string):
    i, parrotType = next((i, parrotType) for i, parrotType in
        enumerate(Types) if match.group(parrotType.groupName) != None)
    counts[i] += 1

def printResults(counts):
  for i, parrotType in enumerate(Types):
    print("{0}: {1}".format(parrotType.description, counts[i]))
  print("Total: " + str(sum(counts)))

def addCounts(counts1, counts2):
  return [x1 + x2 for x1, x2 in zip(counts1, counts2)]

def countOnWebPage(ParrotRegex, Types, pageUrl):
  with urllib.request.urlopen(pageUrl) as request:
    if request.url != pageUrl:
      return None
    ret = [0 for _ in Types]
    countInString(ParrotRegex, Types, ret, request.read().decode("utf-8"))
    return ret

def countIntoQueue(ParrotRegex, Types, pageNumbersQueue, resultsQueue,
    threadUrl):
  processingRequests = True
  while True:
    requestedPageNo = pageNumbersQueue.get()
    if requestedPageNo is None:
      pageNumbersQueue.task_done()
      return

    if processingRequests:
      pageUrl = threadUrl + "/" if requestedPageNo == 1 else \
          "{0}/page-{1}".format(threadUrl, requestedPageNo)
      counts = countOnWebPage(ParrotRegex, Types, pageUrl)
      if counts is None:
        processingRequests = False
      resultsQueue.put(Result(requestedPageNo, counts))
    pageNumbersQueue.task_done()

def poisonQueue(queue):
  for i in range(ProcessPoolSize):
    queue.put(None)

counts = [0 for _ in Types]
threadUrl = sys.argv[1].strip("/")

resultsQueue = mp.Queue()
pageNumbersQueue = mp.JoinableQueue()
highestRequestedPageNumber = 0

def updateJobs(highestPageNoWithResponse):
  global highestRequestedPageNumber
  highestPageNumberToTry = highestPageNoWithResponse + ProcessPoolSize
  for i in range(highestRequestedPageNumber + 1, highestPageNumberToTry + 1):
    pageNumbersQueue.put(i)
  highestRequestedPageNumber = highestPageNumberToTry

for _ in range(ProcessPoolSize):
  counterProcess = mp.Process(target=countIntoQueue, args=(ParrotRegex, Types,
    pageNumbersQueue, resultsQueue, threadUrl))
  counterProcess.start()
updateJobs(0)

while True:
  result = resultsQueue.get()
  if result.counts == None:
    poisonQueue(pageNumbersQueue)
    pageNumbersQueue.join()
    break
  else:
    updateJobs(result.pageNumber)
    counts = addCounts(counts, result.counts)
while not resultsQueue.empty():
  result = resultsQueue.get()
  if result.counts != None:
    counts = addCounts(counts, result.counts)

printResults(counts)

Call both with thread URL:

$ python script.py 'http://www.rpgcodex.net/forums/index.php?threads/parrot-count-m.102131'
Parrot with hat: 25
Normal parrot: 34
Total: 59

You probably shouldn't call them on megathreads, especially the second, unless your research is for the greater good of humanity.

The force is strong within you oh skillful one.
If you don't have a high paying job you sure deserve one.

I haven't looked into the python yet so I only ask about the shell script. Can you chime in on my comments?

{
for i in $(seq 100); do
pageUrl="$threadUrl"/
if [ $i -gt 1 ]; then
pageUrl="$threadUrl"/page-$i
fi

# fetch the page and discard any errors
parrotCount="$(curl -i "$pageUrl" 2>/dev/null |
# feed the page to both sed and grep
# (from sed subshell) save the first line of the html to descriptor 3
tee >(sed '2,$d' >3) | grep -E -o "$parrotRegex" | wc -l)"
# what is 307?
if grep 307 <3 >/dev/null; then
break
fi
totalCount=$(($totalCount + $parrotCount))
done
# what?!
}<3

 

[JRIz]

# fetch the page and discard any errors

It does discard error messages but mostly it hides the progress bar of curl which is printed on stderr.

XenForo redirects you to page 1 of the thread if you request a page that doesn't exist. 307 is the HTTP status code for this redirection. The check is a hack to know what page is the last one.

This whole file descriptor 3 business is actually bullshit. I thought you could somehow make two pipelines at once but in the end >3 just writes to the file 3 (to fd would be >&3). Calling mktemp would make the script better. The '{...}<3' can be removed as well but technically it opens the file 3 for reading on stdin within the braces which is unnecessary, of course.

 

[a Goat]

Hidden parrot agenda incoming

 

[Ulminati]

your insightful comentary has persuded me. just finished installing freya - don't give me bs about <insert distro here> distribution being better. it looked pretty.

edit: installing python. linux's tagline should be "by autists for autists". they spend all this time on interface and I still have to pull up a terminal to get anything meaningful done.

ffs, just make it so I can double click to run/install things. how hard is that?

You mean like the software manager in mint? (I know you said don't give you crap about distros, but it's specifically aimed at people who don't want to pull up a terminal to get anything meaningful done).

Or

> sudo apt-get update
> sudo apt-get install python

Throw in
> sudo apt-get upgrade
while you're at it

 

[a cut of domestic sheep prime]

You mean like the software manager in mint? (I know you said don't give you crap about distros, but it's specifically aimed at people who don't want to pull up a terminal to get anything meaningful done).

Or

> sudo apt-get update
> sudo apt-get install python

Throw in
> sudo apt-get upgrade
while you're at it

Maybe, yes. I'm not a big fan of the terminal. Don't get me wrong, DOS was my first OS, so it doesn't scare me, but I'm a visual person and hate having to mess with it. I want things to work fast and without me having to devote any thought to it because I'm a busy parrot-counting individual and can't be bothered with it.

Also, before installing Freya I had zero clue about the terminal or "sudo anything" for that matter. I may have learned it briefly a decade ago when I messed with Linux, but I haven't touched it since, so I've forgotten pretty much everything about it.

 

[LJ40]

No, I will join the Linux help forum and post parrot swastikas all day, so I don't have to hear retards like lambchop bitch at me because I actually mentioned something in my post that was on topic.

It might help to practice a bit, because that is not a swastika.

 

[a cut of domestic sheep prime]

Yeah, I noticed it too, but I was just going to let it go...

 

[Mustawd]

It might help to practice a bit, because that is not a swastika.

A swastika can be easily made with python. But I'm not going to tell you how.