[Wiesbaden] Data

[jklmnn]

Wiesbaden also has parking data available at https://www.wiesbaden.de/leben-in-wiesbaden/verkehr/verkehrsinfos/parken.php. The raw data seems to be at https://wi.memo-rheinmain.de/wiesbaden/parkliste.phtml?order=carparks. While it looks quite ugly to parse it seems to have all the data we need.

[kiliankoe]

While it looks quite ugly to parse

Lol, this is far better than most others 😄

[jklmnn]

But only with whitespace correction :smile:

[AugustQu]

Hi,

here I am. From what I understand: first I deliver the coordinates of the parking-lots given in the URL above.

So this may look like: static_data["City I"] = {'latitude': "50.08121", 'longitude': "8.23637", 'adresse': "Schwalbacher Str. 41-43"}

Am I correct?

And a question: this parking-lot is closed on sundays/holidays and opened on monday to saturday from 7:00 to 20:00. How do I enter this information?

PS: I've taken this file as an example: https://github.com/Bytespeicher/pyParkingLotEF2Json/blob/master/pyParkingLotEF2Json.py

[AugustQu]

I've seen a lot of py-files. Looking at the code I found:

• create a loop over all parking-lots (14 for WI)
• search for the name (td-Element)
• next is the 'Freie Stellplätze'
• finally: return the name together wit the numbers.

Am I right?

PS: I'm new to python.

[jklmnn]

Hi,

here I am. From what I understand: first I deliver the coordinates of the parking-lots given in the URL above.

So this may look like: static_data["City I"] = {'latitude': "50.08121", 'longitude': "8.23637", 'adresse': "Schwalbacher Str. 41-43"}

Am I correct?

And a question: this parking-lot is closed on sundays/holidays and opened on monday to saturday from 7:00 to 20:00. How do I enter this information?

PS: I've taken this file as an example: https://github.com/Bytespeicher/pyParkingLotEF2Json/blob/master/pyParkingLotEF2Json.py

The project you are referring to seems to be outdated. Also it doesn't use our API.

The files relevant for you reside in park_api/cities. I'd recommend you to take a look at Sample_City.py and Sample_City.geojson. You could copy those files to Wiesbaden.py and Wiesbaden.geojson respectively and then edit them. Wiesbaden should then automatically appear in your local instance.

For parsing we use beautifulsoup4. I'd advise you to do the same since it is easily powerful enough for the task and already a dependency. About the data the relevant table is accessible via the xpath /html/body/table/tbody/tr[3]/td/div/table/tbody and then iterate over all trs except the first one (which is the table header) and get the data from the tds.

Another far simpler, but important and time consuming task would be filling the geojson file which needs to be done by hand but only once. The most important information are the coordinates.

[AugustQu]

Hi,

I'm not only new to Python but also new to Github.

I copied the whole project as a ZIP-file, expanded it and modified the file Sample_city.geojson adding all the 14 parking-lots of Wiesbaden. Finally I saved it as Wiesbaden.geojson, locally on my PC.

Some questions: #1: in the 'features' of this file coordinates have to be entered. I entered the coordinates of the city town hall. Is this correct? #2: in the 'properties' I have to enter an URL. Here I entered "https://www.wiesbaden.de/leben-in-wiesbaden/verkehr/verkehrsinfos/parken.php". Correct? Also as a source I entered: "https://wi.memo-rheinmain.de/wiesbaden/parkliste.phtml?order=carparks". Correct? #3: there is one parking lot that has 2 entries for cars. Can this be handled? Or shall I omit the second entry?

[jklmnn]

I'd recommend to check out the GitHub Hello World tutorial and the git tutorial (while GitHub builds upon git, git is a free version control system). There also is a collection of resources that looks like a good start.

As for the general workflow, you should fork this repository, do your changes on your on fork and the create a pull request with your changes for us to review.

1. The data includes a list of features with the first being the city itself and all others being parking lots. The coordinates of the first feature should be either the geographical center of the city or the geographical center of the distribution of parking lots. We usually take the coordinates of large map providers for this.
2. Yes that's both correct. The url field denotes the public accessible location of the data (city homepage, etc) and the source denotes the actual source of the information that the parser will receive.
3. Do you mean entries in how you enter the lot or data entries? In the first case we only use one address and coordinate. In the second case I'm not sure how it would look like.

[AugustQu]

Hi,

thanks for the link to 'hello world' on GitHub. I will look at it.

Now I have the file Wiesbaden.geojson ready I will check it again, polish it a bit and then open a Pull Request.

In parallel I will try to work on the file Wiesbaden.py.

[jklmnn]

Thanks! I added a branch wiesbaden so when you open the pull request please select this as the target branch since we only allow finished features on master. When everything works wiesbaden will then be merged into master.

[AugustQu]

playing with Python I got this result: my code:

table=soup.select('table') td = table[2].find_all("td") i=0 while i < len(td): print('name:',td[i+1].text.strip()) print('frei=', int(td[i+2].text.split()[0])) print('total=', int(td[i+2].text.split()[2])) i += 5

the result, first entry only:

('name:', u'City I') ('frei=', 137) ('total=', 165)

hint: I added the html statically for my test.

So I have to add these informations to the data-object and I should be done.

Looking into the Sample_City.py I find these lines:

    data["lots"].append({

        "name": lot.name,
        "free": lot_free,
        "total": lot_total,
        "address": lot.address,
        "coords": lot.coords,
        "state": state,
        "lot_type": lot.type,
        "id": lot.id,
        "forecast": False,

The first 3 lines are the information given above, the rest comes from the geojson-file looked into via the name. Correct?

PS: I have to check for state and lot.id.

[AugustQu]

I added this to my code:

stand=soup.select('span')
last_updated_date=stand[0].text.strip().split()[1] last_updated_time=stand[0].text.strip().split()[2] print("last updated=", last_updated_date, " ", last_updated_time)

which results in this line: ('last updated=', u'07.04.2019', ' ', u'11:16')

[AugustQu]

I just added the files Wiesbaden.geojson and Wiesbaden.py but I'm not sure that I did it correctly.

In the Python-script I added 4 TODOs which describe lines to look for:

• the handling of last_updated
• the state (twice)
• the lot.id which I do not know

Can you please check these lines and also the whole script. Thanks.

[AugustQu]

one error I made: I loaded the file Sample_city.geojson, replaced the contents with my file (Wiesbaden.geojson) and saved it as Wiesbaden.geojson. Instead I should have created a new file via the button.

Same for the py-file.

Sorry.

[jklmnn]

Sorry for my late reply.

The first 3 lines are the information given above, the rest comes from the geojson-file looked into via the name. Correct?

Yes that's true.

There are some issues about the pull requests. One is they're opened against master which we won't merge until everything works perfectly. But you can easily choose the wiesbaden branch with the dropdown on the left:

The second thing is I'd suggest you only make a single pull request with all your changes. This helps us to directly comment on all parts of your code to help you better. By pushing to the branch of the single pull request it will be updated and we always will see the changes. Furthermore having all changes on a single branch helps us checking it out and testing it more easily.

[AugustQu]

what do I have to do now? Add a new Pull-request against the Wiesbaden-branch?

Or do you handle this situation?

[jklmnn]

Yes, you have to open a new pull request against the wiesbaden branch.

[AugustQu]

OK. I added the files Wiesbaden.py and Wiesbaden.geojson in the cities-directory of the Wiesbaden-branch and did a Pull-request. I hope this is correct now.

Unfortunately I added the file Wiesbaden.py to the wrong directory in the first try so please delete this file. And also it looks like I did overwrite the file Sample_city.py with the code for Wiesbaden. Can I correct this? Or do you simply reject the 'modifications'?

[jklmnn]

Thanks, that looks better. Since the overwritten Sample_City.py file is in its own commit it doesn't pose a problem as I can omit it when merging. The current process will look as follows: I'll review your code, probably request some changes which you then can push to the pull request branch. When we're at the point that the code is clean and working we'll merge the pull request.

[AugustQu]

I will look at the status of the parking-lot over the weekend, then make some changes to the code.

I've seen that the data on wiesbaden.de showed the status of 2 parking-lots as closed, I saw this last night. I will try to find out how they do it.

[AugustQu]

it looks like there are only minor modifications needed. So I will look for the status, make the needed changes plus the already mentioned modifications and deliver them together.

I gues the next version will come on monday.

One hint: the word Uhr is included in the following line taken from the file Sample_city.py:

    "last_updated": convert_date(last_updated, "%d.%m.%Y %H:%M Uhr"),

[jklmnn]

One hint: the word Uhr is included in the following line taken from the file Sample_city.py

This is of course just an example and needs to be adapted to the actual data.

[AugustQu]

OK, I got it.

Uploading my code to the branch wiesbaden I did encounter 2 problems:

• I do not see the old files any more

• so I tried to create new files but could not save them because I have no write access

What can I do now?

[AugustQu]

there is this: AugustQu/ParkAPI there is a wiesbaden-branch where I can find my files.

I will add my latest modifications to these files and then make another pull-request.

[AugustQu]

so I hope I did it correctly.

[AugustQu]

I made some modifications to the py-script and will upload this file later. These modifications were of the type beautifying.

But I do see one problem: there is a parking-lot with the name Coulinstraße. On the memo-rheinmain-page it appears as: Coulinstraße, in the geojson-file I added it as Coulinstrasse. In the py-script this will be handled as: parking_name = td[i+1].text.strip() lot = geodata.lot(parking_name) so the name is extracted from the memo-rheinmain-page and this is used for the search in the geojson-file. This will result in no match.

How can I handle this? Enter the name in the geojson-file as Coulinstraße?

[AugustQu]

I found a typing-error: in the geojson-file the name of a parking-lot is written as: Rhein Main in the memo-rheinmain-site it is written as: RheinMain

I will fix this.

Put this aside: does the script work?

[jklmnn]

Sorry for my late reply, I'm quite busy at the moment.

so I hope I did it correctly. It seems you pushed your changes to the already existing pull request, so yes that's correct. I will review it when I find time.

About ß in street names, they should be usually encoded in UTF-8 that supports those characters. Although if the page doesn't have the correct encoding they will be erroneous. I think we have a solution for this to at least convert those characters to ASCII (@kiliankoe do you remember how we fixed that in other cities?).

Put this aside: does the script work?

Same as above, I can tell when I find time to review (and test ofc.).

[AugustQu]

Sorry for my late reply, I'm quite busy at the moment.

OK. The usual thing: I created some lines and want to see the result immediately.....

I've jut added some minor modifications to the two files. One ist the name 'Coulinstraße' in the geojson-file and the other is a bit of cleaning the code in the py-script.

[jklmnn]

Thanks for the changes, I reviewed them. There are two stray lines in the geojson that shouldn't be there. If you remove them I think it's ready to be merged.

One last thing. There is the active_support flag in the json which decides if the city is enabled by default in the apps. If you want the city to be enabled by default I'd suggest you set it to true.

If the website changes and the parser breaks (happens usually rarely) we will either try to fix it or (if the fix requires more work) we probably set active_support to false again unless someone fixes it. That what the flag is for.

[kiliankoe]

I think we have a solution for this to at least convert those characters to ASCII (@kiliankoe do you remember how we fixed that in other cities?).

We have util.remove_special_chars, but that doesn't appear to be sophisticated enough for this specific use-case. There must be a default and sane way to deal with this, probably even built-in to Python.