[Python] Program not working correctly

[[null]]

Hello fellow iNinjas! I got another python problem for you guys to try to figure out. I'm working on a little project I call 'Cthulhu'. Basically, the idea is a clone of harvester written in python. I started working on it for two reasons: one, you could get a list from Harvester and one from Cthulhu and combine them for a better chance of a successful brute force, or if your like me and don't like cluttering up your 8gb of space with another programming language (perl). Anyway, I've run into a little problem with it. Whenever I try to run it, first it works just fine, but then I get this: 


And when I look in the filesystem with iFile, URL.txt and source.txt where created, but they are totally empty. 


Here is the source for linkex.py:

Code: [Select]

# get_links by Jabba! Modified for use in Cthulhu by [null]

import re
import sys
import urllib
import urlparse
from BeautifulSoup import BeautifulSoup
import os

print "Creating URL file..."
f = open("URL.txt", "w")
class MyOpener(urllib.FancyURLopener):
    version = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15'

def process(url):
    myopener = MyOpener()
    page = urllib.urlopen(url)
    page = myopener.open(url)
    text = page.read()
    page.close()
    soup = BeautifulSoup(text)
    for tag in soup.findAll('a', href=True):
        tag['href'] = urlparse.urljoin(url, tag['href'])
        print "Writing URLs to file..."
        f.write(tag['href'])
        os.system("python sourceget.py")
# process(url)

def main():
    if len(sys.argv) == 1:
        print "Jabba's Link Extractor v0.1"
        print "Usage: %s URL [URL]..." % sys.argv[0]
        sys.exit(-1)
    # else, if at least one parameter was passed
    for url in sys.argv[1:]:
        process(url)
# main()

if __name__ == "__main__":
    main()
And here is the source for the other program, sourceget.py:

Code: [Select]

 # Get source code from URL list

import urllib2
import os

print "Creating source file..."
l = open("source.txt", "w")
l.close()
l = open("source.txt", "a")
line = 1
var = 1
print "Reading from URL file..."
url = open("URL.txt", "r").readlines()
while var == 1:
     try: 
          print "Getting source of line ", line
          usock = urllib2.urlopen(url[line])
          data = usock.read()
          l.write(data)
          line = line + 1
     except IndexError:
          print "Done!"
#usock.close()Any help would be much appreciated!

--[null]

[8BitAce]

A few things:

You need to break the

Code: [Select]

while var==1 loop when it reaches an index error by setting var to something else. Right now it's an infinite loop.

You currently have

Code: [Select]

usock.close() commented out, that socket will never close!

Try removing the

Code: [Select]

os.system("python sourceget.py") line and see if URL.txt is ever written to before the parsing removes the lines.

[[null]]

Aaaargh infinite loop! Lol, that was a stupid mistake. And I forget why I commented that usock.close() out. Anyway, I'll try out why you said. Thanks!

[[null]]

Alright, so I did what you said and linkex.py works just fine! But when I run sourceget.py (after removing the infinite loop) all I get is

Code: [Select]

Creating source file...
Reading from URL file...
Getting source from line 1
Done!
But when I look in the directory, again no luck.

[8BitAce]

What URL are you giving it?

[[null]]

http://www.google.com

[[null]]

I'm gonna try runnin it on my tablet, which has python 3.

[[null]]

Well, I tried it, and no luck. Any ideas?

[8BitAce]

Okay, got it to work. Here's the changes:

In linkex.py change line 25 to:

Code: [Select]

f.write(tag['href'] + '\n')
Otherwise it spits out as one big line

Also I think you want to take

Code: [Select]

os.system("python sourceget.py")
Out of the loop as it only needs to run once.

And more importantly, add

Code: [Select]

f.close()At the end of the process() function (outside the loop!). Currently the system never releases the lock on the file, which is why it was empty.


Then for sourceget.py, change line 10 to:

Code: [Select]

var = 0Since python is a a zero-indexing language.


Now on Windows, I have to run each script separately, but it should work on iOS.

[[null]]

Thanks a ton 8bit! +1! I can't wait to get this to work!

[8BitAce]

Happy to help! Let us know how it works

[Alex47]

If its a clone of harvester, will it do 'exactly' the same thing? Because harvester for me doesnt get a password list, it just grabs every single file and I find that more useful. Will this do that? If it does can you consider support for Tor sites?

[[null]]

It doesn't? Sorry, I've never used the Harvester before because I only have a tiny 8gig iPod. My understanding was that it downloaded the source for the entire website, and then went through and found possible passwords and usernames, then made a wordlist out of it. The TOR idea is great though! I think I might implement that!

[Alex47]

Ok thanks, and I think I just chose bad sites :/

Good luck!