Bonjour !

You must get the key down (south key). It is in the same village where you got the key left (west key). There is a house with a woman and a child inside. Talk to the woman. Then give the woman a milk candy (bonbon au lait). She will give the key down (south key) to you. Then you can return to the city and use the key.

I hope this helps. If you have more questions, please let me know. Good luck!