Unfortunately after installing the batteries there were no way to turn it off.
I have opened up with: "worst case I will put a switch to it" in my mind.
The circuit inside is simple: basically an SOT-23-6 IC marked with 819L 48 a power P fet in SO8 package and some current limiting resistors:
After doing some google fu I ended up on this website:
http://go-radio.ru/remont-nalobnogo-fonarya.html
It turned out that the IC is a Shenzhen Fuman Elec FM2819 The datasheet is only available in chinese, but pushing through the google translate did not helped either.
I have noticed one thing: there were an unpopulated 0603 place parallel with the pushbutton. The russian blog mentioned above that he have had found a cap in the box fallen from the same place. I have followed his advice put a 0603 100nF cap there and now the lamp works as expected.