I don't know if the namespace settings matter...

rich_is_bored wrote:

Now this one I dunno. I'm not sure what the default namespace should be nor what the location of index.php is. I'm just guessing.

I believe iceheart configured Apache to rewrite the urls - for example, the homepage is http://www.modwiki.net/wiki/Main_Page .

This bot seems to want an index.php, so it is probably using the old syntax. Seeing how http://www.modwiki.net/w/index.php?title=Main_Page works, I would use the default value '/w/index.php'.

That was it. Thanks a ton.

I've tested it a few times now and it works perfectly. It's very easy to use.

Now I just have to do a few extensive test runs without uploading anything and iron out bugs. I should have something by Friday.

Somehow, I seem to have missed this thread, but it looks like you worked it all out without me

.

I'd just like to take a moment and speak yet again of the test wiki, found at:

http://test.modwiki.net/w/index.php/Main_Page

It is completely identical to the live wiki except for the URL rewriting. Use it for testing anything that might potentially break the site or cause problems that will take lots of time and effort to reverse (in mediawiki, it is extremely difficult to do such things, but hey, you might manage to do it

I thought about this a little and maybe, in addition to {{stub}}, we should also have a {{bot}} template, saying something like "This page is auto-generated by a bot from declaration files blah blah blah please add human information if you have it" at the bottom. Maybe a special "auto generated" category as well?

Comments?

BTW. The "bot" flag, which I have again set on the bot user, does nothing except prevent edits from showing up in change lists (unless you click the "show bots" link).

Indeed, it might be usefull to have a "Bot" template and an "Auto-Generated" category associated with this template. (I propose "Auto-generated" because it might be usefull to reserve the name "Generated" for ET:QW's binary file formats.)

However, I would put the "bot" template at the top seeing how some articles might turn out to be fairly long.

If nobody has a problem with this, I'll make the template tomorrow.

I've added some useful information to PyBot's user page...

http://www.modwiki.net/wiki/User:PyBot

As far as templates and categories are concerned, go ahead and implement those changes using the monster_zombie_fat article as an example and I'll be sure to incorporate whatever changes you make into the final script.

Starting last night I set the bot to work on Doom 3 and Quake 4. But that'll be more evident to you when you see the "3000+ articles" statistic on the main page.

It takes the bot about 3 hours to upload the content for each game (It "sleeps" for about 10 seconds between articles) so I suppose it's a good thing I'm doing each game separately.

There are a few links in the respective lists that don't seem to have a corresponding article. I'm pretty sure that's a fault of the main page rather than the script. But I'll sort it out later.

I'll set the bot to work on Prey before I go to sleep tonight.

As for ETQW, that will have to wait a bit. It looks like the def files are compressed and I'll need the SDK to obtain class inheritance information.

The Entity listing is fine for now, but if we're going to regenerate it to make it tree-like, shouldn't we make different pages for different games? I believe there was some discussion why we did it like that for CVars, commands and script events, the same applies here.

An extra reason to do so is because you can see from which game what entity is if you look at the "What links here" special page.

That makes sense. I don't see why we couldn't do that.

Just to keep everyone in the loop and let you know I'm still at it, I've started encapsulating the code into classes so that from this point forward we have a really modular base to use for more purposes than just the wiki.

For example, what if I wanted to know how the lights in Doom 3 differed from Quake 4? You could write python code like this...

Code:

import idTech4

def test()
   
   # Load Data for each game
   doom3 = idTech4.gameData('E:\\Doom3 Assets\\def', 'E:\\D3_SDK\\src')
   quake4 = idTech4.gameData('E:\\Quake4 Assets\\def', 'E:\\Q4_SDK\\source')

   doomlight = doom3.getKeys('light')
   quakelight = quake4.getKeys('light')
   
   if len(doomlight) != len(quakelight):
      for x in quakelight:
         if x not in doomlight:
            print x, 'is not present in Doom3'

It's probably not the best example but hopefully you get the idea.

Here's the module as it is right now...

Code:

import fileinput
import os
import re

class gameData:
    def __init__(self, basedir, sourcedir):
        '''Initialize gameData class'''
    
        self.entdef = re.compile(r'entityDef\s(\w*)\s{([^}]*)}')
        self.kvpair = re.compile(r'\n\s{0,10}"([^"]*)"\s*"([^"]*)"')
        self.prefix = re.compile(r'(\w+)[\x20\t](\w+)')
        self.clsmethod = re.compile(r'(\w*)::\w*\([^)]*\)\s*(?:\w*\s)?{([^\xBF]*?(?<=\n))}')
        self.spnarg = re.compile(r'spawnArgs\.[^(]*\(\s"(\w*)')
        self.clsdec = re.compile(r'CLASS_DECLARATION\(\s([^,]*),\s([^)]*?)\s\)')
        
        # TODO: Validate basedir
        self.basedir = basedir
        # TODO: Validate sourcedir
        self.sourcedir = sourcedir
        self.data = dict()
        
        self.loadEntities()
        self.loadClasses()
        
    def loadEntities(self):
        '''Load entity declarations into class'''
    
        print ' Loading entity declarations ...',
        filelist = []
        for (path, dname, fnames) in os.walk(self.basedir):
            for fn in fnames:
                if fn.endswith('.def'):
                    filelist.append(os.path.join(path, fn))
        datastring = str()
        for line in fileinput.input(filelist):
            datastring = datastring + line
        match = re.findall(self.entdef, datastring)
        for x in match:
            name = x[0]
            kvpairs = dict(re.findall(self.kvpair, x[1]))
            self.data[name] = kvpairs
            self.data[name].setdefault('sdkclass', False)
        print 'done.'
        
    def loadClasses(self):
        '''Load source code data into class'''
    
        print ' Loading source code classes ...',
        filelist = []
        for (path, dname, fnames) in os.walk(self.sourcedir):
            for fn in fnames:
                if fn.endswith('.cpp') or fn.endswith('.h'):
                    filelist.append(os.path.join(path, fn))
        datastring = str()
        for line in fileinput.input(filelist):
            datastring = datastring + line
        match = re.findall(self.clsmethod, datastring)
        for x in match:
            name = x[0]
            kvpairs = list(re.findall(self.spnarg, x[1]))
            temp = dict()
            for y in kvpairs:
                temp[y] = None
            self.data[name] = temp
            self.data[name].setdefault('sdkclass', True)
        match = re.findall(self.clsdec, datastring)
        for x in match:
            name = x[1]
            inherit = x[0]
            if name not in self.data:
                self.data[name] = {}
            self.data[name].setdefault('inherit',inherit)
            self.data[name].setdefault('sdkclass', True)
        print 'done.'
        
    def isClass(self, name):
        '''True or False / Class or Entity'''
        return self.data[name].get('sdkclass')
        
    def getInherit(self, name, templist = None):
        '''Returns list cooresponding to inheritance chain for specified entity/class'''
        if templist == None:
            templist = []
        templist.append(name)
        if name in self.data:
            if 'inherit' not in self.data[name]:
                if 'spawnclass' not in self.data[name]:
                    pass
                else:
                    return self.getInherit(self.data[name]['spawnclass'], templist)
            else:
                return self.getInherit(self.data[name]['inherit'], templist)
        return templist
        
    def getKeys(self, name):
        '''Returns all key/value pairs for specified entity/class'''
        keys = []
        for x in self.getInherit(name):
            for y in self.data[x].keys():
                if y not in keys:
                    keys.append(y)
        keys.sort()
        keys.remove('sdkclass')
        return keys
        
    def getPrefix(self, name):
        '''Returns all prefixed key/value pairs for specified entity/class'''
        keys = []
        for x in self.getKeys(name):
            for y in self.prefix.findall(x):
                if y[0] not in keys:
                    keys.append(y[0])
        keys.sort()
        return keys
        
    def getValue(self, name, key):
        '''Returns value of key for specified entity/class'''
        for x in self.getInherit(name):
            if key in self.data[x]:
                return self.data[x].get(key)
                break
                
    def getDesc(self, name, key = None):
        '''Returns description of entity or key if specified.'''
        if key == None:
            for x in self.getInherit(name):
                if 'editor_usage' in self.getKeys(x):
                    return self.getValue(x, 'editor_usage')
                    break
        else:
            for x in self.getInherit(name):
                for y in self.getKeys(x):
                    if y.startswith('editor_var'):
                        for z in self.prefix.findall(y):
                            if z[1].startswith(key):
                                return self.getValue(name, y)

def test():
    print 'This is a module, not a stand alone program.'

if __name__ == '__main__':
    test()

My goal here is to get it to the point where I can write a module to create the tree diagrams needed for the main page of each entity listing. I tried it with my previous code and I was running into a lot of issues.

And so far it's paid off...

Note, this is just a snippet of the generated tree. When you zoom out to view the entire tree it starts looking like a vertical sound wave.

Interesting... And again, great job!