schmaker,

Hello guys,

I'm getting #PHP process killed by #OOM killer lately and am not able to debug what and why is causing it, only thing i know at this moment is, that it seems to be related to #Friendica worker (no other webpages are running at my VPS.

Errors in friendica.log and httpd.log are repeated over and over again, I have tons of these.

Friendica log```
2024-01-14T12:14:05Z worker [ALERT]: Fatal Error (E_ERROR): Allowed memory size of 268435456 bytes exhausted (tried to allocate 50348032 bytes) {"code":1,"message":"Allowed memory size of 268435456 bytes exhausted (tried to allocate 50348032 bytes)","file":"/srv/http/schmaker.eu/src/Database/Database.php","line":568,"trace":null} - {"file":null,"line":null,"function":null,"request-id":"65a3d00a94ba8","stack":"ErrorHandler::handleFatalError","uid":"0e21ff","process_id":524483}
2024-01-14T12:14:19Z app [NOTICE]: created item {"post-id":1746909,"uid":0,"network":"apub","uri-id":7056319,"guid":"52acf9cd-d5329c5bc33553a3-16fb1418"} - {"file":"Item.php","line":1334,"function":"insert","request-id":"65a3d01b052b3","stack":"Item::insert (1172), Processor::postItem (783), Receiver::routeActivities (747), Receiver::processActivity (169), Receiver::processInbox (101), Inbox::post (233), BaseModule::run (109), BaseApi::run (43), Inbox::run (711), App::runFrontend (46)","uid":"1ddf6c","process_id":521334}
2024-01-14T12:14:19Z app [NOTICE]: created item {"post-id":1746910,"uid":2,"network":"apub","uri-id":7055024,"guid":"2cc3d87f-74a5bdcec9bb28f3-597a3200"} - {"file":"Item.php","line":1334,"function":"insert","request-id":"65a3d01b052b3","stack":"Item::insert (1874), Item::storeForUser (1740), Item::storeForUserByUriId (1252), Processor::hasParents (1162), Processor::postItem (783), Receiver::routeActivities (747), Receiver::processActivity (169), Receiver::processInbox (101), Inbox::post (233), BaseModule::run (109), BaseApi::run (43), Inbox::run (711), App::runFrontend (46)","uid":"1ddf6c","process_id":521334}
2024-01-14T12:14:19Z app [NOTICE]: Inserted missing parent post {"stored":1746910,"uid":2,"parent":"https://mastodonczech.cz/users/mkyral/statuses/111754066555820024"} - {"file":"Processor.php","line":1255,"function":"hasParents","request-id":"65a3d01b052b3","stack":"Processor::hasParents (1162), Processor::postItem (783), Receiver::routeActivities (747), Receiver::processActivity (169), Receiver::processInbox (101), Inbox::post (233), BaseModule::run (109), BaseApi::run (43), Inbox::run (711), App::runFrontend (46)","uid":"1ddf6c","process_id":521334}
2024-01-14T12:14:19Z app [NOTICE]: created item {"post-id":1746911,"uid":2,"network":"apub","uri-id":7056319,"guid":"52acf9cd-d5329c5bc33553a3-16fb1418"} - {"file":"Item.php","line":1334,"function":"insert","request-id":"65a3d01b052b3","stack":"Item::insert (1172), Processor::postItem (783), Receiver::routeActivities (747), Receiver::processActivity (169), Receiver::processInbox (101), Inbox::post (233), BaseModule::run (109), BaseApi::run (43), Inbox::run (711), App::runFrontend (46)","uid":"1ddf6c","process_id":521334}
2024-01-14T12:14:28Z app [NOTICE]: no data for session {"session_id":"918db4sbltu4uuboiu7vgir1u1","uri":"/robots.txt"} - {"file":"Database.php","line":78,"function":"read","request-id":"65a3d0247394c","stack":"Database::read, session_start (54), Native::start (150), Dice::Dice{closure} (96), Dice::create (247), Dice::Dice{closure} (132), Dice::Dice{closure} (96), Dice::create (247), Dice::Dice{closure} (132), Dice::Dice{closure} (96), Dice::create (92), DI::app (42)","uid":"21ebe6","process_id":521020}
2024-01-14T12:14:46Z app [NOTICE]: created item {"post-id":1746912,"uid":0,"network":"apub","uri-id":7056320,"guid":"f7dd981d-1265-a37e-95d3-45a258252183"} - {"file":"Item.php","line":1334,"function":"insert","request-id":"65a3d0369bf29","stack":"Item::insert (1172), Processor::postItem (708), Processor::createActivity (843), Receiver::routeActivities (747), Receiver::processActivity (169), Receiver::processInbox (101), Inbox::post (233), BaseModule::run (109), BaseApi::run (43), Inbox::run (711), App::runFrontend (46)","uid":"ced645","process_id":511968}
2024-01-14T12:14:46Z app [NOTICE]: created item {"post-id":1746913,"uid":2,"network":"apub","uri-id":7056320,"guid":"f7dd981d-1265-a37e-95d3-45a258252183"} - {"file":"Item.php","line":1334,"function":"insert","request-id":"65a3d0369bf29","stack":"Item::insert (1172), Processor::postItem (708), Processor::createActivity (843), Receiver::routeActivities (747), Receiver::processActivity (169), Receiver::processInbox (101), Inbox::post (233), BaseModule::run (109), BaseApi::run (43), Inbox::run (711), App::runFrontend (46)","uid":"ced645","process_id":511968}
2024-01-14T12:14:49Z app [NOTICE]: no data for session {"session_id":"0ahqk6onff097rtsfs76uk85vu","uri":"/profile/netvoyne?tag=Zaporozhye&page=7"} - {"file":"Database.php","line":78,"function":"read","request-id":"65a3d0395754d","stack":"Database::read, session_start (54), Native::start (150), Dice::Dice{closure} (96), Dice::create (247), Dice::Dice{closure} (132), Dice::Dice{closure} (96), Dice::create (247), Dice::Dice{closure} (132), Dice::Dice{closure} (96), Dice::create (92), DI::app (42)","uid":"28fdfc","process_id":521334}
2024-01-14T12:14:50Z app [NOTICE]: no data for session {"session_id":"f1176v7k02rugm0aqrpgo7l8eg","uri":"/display/c83e3896-1964-2eb2-f5bf-0c2147726828"} - {"file":"Database.php","line":78,"function":"read","request-id":"65a3d03a79379","stack":"Database::read, session_start (54), Native::start (150), Dice::Dice{closure} (96), Dice::create (247), Dice::Dice{closure} (132), Dice::Dice{closure} (96), Dice::create (247), Dice::Dice{closure} (132), Dice::Dice{closure} (96), Dice::create (92), DI::app (42)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on null {"code":2,"message":"Trying to access array offset on null","file":"/srv/http/schmaker.eu/view/smarty3/compiled/d0/de/3f/d0de3f3fd98d576056fb02f629b93c7eb0442beb_2.file.threaded_conversation.tpl.php","line":46} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (46), content_65894a0656ce28_18927795 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (238), Smarty_Internal_TemplateBase::_execute (116), Smarty_Internal_TemplateBase::fetch (100), FriendicaSmartyEngine::replaceMacros (84), Renderer::replaceMacros (593), Conversation::render (278), Display::getDisplayData (165), Display::content (250), BaseModule::run (711), App::runFrontend (46)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on null {"code":2,"message":"Trying to access array offset on null","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":506} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (506), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48), content_65894a0656ce28_18927795 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (238), Smarty_Internal_TemplateBase::_execute (116), Smarty_Internal_TemplateBase::fetch (100), FriendicaSmartyEngine::replaceMacros (84), Renderer::replaceMacros (593)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on null {"code":2,"message":"Trying to access array offset on null","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":521} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (521), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48), content_65894a0656ce28_18927795 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (238), Smarty_Internal_TemplateBase::_execute (116), Smarty_Internal_TemplateBase::fetch (100), FriendicaSmartyEngine::replaceMacros (84), Renderer::replaceMacros (593)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on null {"code":2,"message":"Trying to access array offset on null","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":539} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (539), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48), content_65894a0656ce28_18927795 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (238), Smarty_Internal_TemplateBase::_execute (116), Smarty_Internal_TemplateBase::fetch (100), FriendicaSmartyEngine::replaceMacros (84), Renderer::replaceMacros (593)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on null {"code":2,"message":"Trying to access array offset on null","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":554} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (554), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48), content_65894a0656ce28_18927795 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (238), Smarty_Internal_TemplateBase::_execute (116), Smarty_Internal_TemplateBase::fetch (100), FriendicaSmartyEngine::replaceMacros (84), Renderer::replaceMacros (593)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on false {"code":2,"message":"Trying to access array offset on false","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":798} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (798), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48), content_65894a0656ce28_18927795 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (238), Smarty_Internal_TemplateBase::_execute (116), Smarty_Internal_TemplateBase::fetch (100), FriendicaSmartyEngine::replaceMacros (84), Renderer::replaceMacros (593)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on false {"code":2,"message":"Trying to access array offset on false","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":1103} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (1103), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48), content_65894a0656ce28_18927795 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (238), Smarty_Internal_TemplateBase::_execute (116), Smarty_Internal_TemplateBase::fetch (100), FriendicaSmartyEngine::replaceMacros (84), Renderer::replaceMacros (593)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on null {"code":2,"message":"Trying to access array offset on null","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":506} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (506), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48), content_65894a0656ce28_18927795 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (238)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on null {"code":2,"message":"Trying to access array offset on null","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":521} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (521), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48), content_65894a0656ce28_18927795 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (238)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on null {"code":2,"message":"Trying to access array offset on null","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":539} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (539), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48), content_65894a0656ce28_18927795 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (238)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on null {"code":2,"message":"Trying to access array offset on null","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":554} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (554), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48), content_65894a0656ce28_18927795 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (238)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on false {"code":2,"message":"Trying to access array offset on false","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":798} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (798), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48), content_65894a0656ce28_18927795 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (238)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on false {"code":2,"message":"Trying to access array offset on false","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":1103} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (1103), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48), content_65894a0656ce28_18927795 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (238)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on null {"code":2,"message":"Trying to access array offset on null","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":506} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (506), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on null {"code":2,"message":"Trying to access array offset on null","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":521} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (521), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on null {"code":2,"message":"Trying to access array offset on null","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":539} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (539), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on null {"code":2,"message":"Trying to access array offset on null","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":554} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (554), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48)","uid":"9ecf0a","process_id":516104}
2024-01-14T12:14:50Z app [WARNING]: E_WARNING: Trying to access array offset on false {"code":2,"message":"Trying to access array offset on false","file":"/srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php","line":798} - {"file":"smarty_internal_errorhandler.php","line":112,"function":"handleError","request-id":"65a3d03a79379","stack":"ErrorHandler::handleError (112), Smarty_Internal_ErrorHandler::handleError (798), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (384), Smarty_Internal_Template::_subTemplateRender (1158), content_65894a1d891d04_34538026 (123), Smarty_Template_Resource_Base::getRenderedTemplateCode (114), Smarty_Template_Compiled::render (217), Smarty_Internal_Template::render (386), Smarty_Internal_Template::_subTemplateRender (48)","uid":"9ecf0a","process_id":516104}


  
httpd error log```
[Sun Jan 14 13:10:37.773690 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 506
[Sun Jan 14 13:10:37.773743 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 521
[Sun Jan 14 13:10:37.773791 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 539
[Sun Jan 14 13:10:37.773874 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 554
[Sun Jan 14 13:10:37.773940 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on false in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 798
[Sun Jan 14 13:10:37.773990 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on false in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 1103
[Sun Jan 14 13:10:37.774111 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 506
[Sun Jan 14 13:10:37.774182 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 521
[Sun Jan 14 13:10:37.774240 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 539
[Sun Jan 14 13:10:37.774291 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 554
[Sun Jan 14 13:10:37.774364 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on false in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 798
[Sun Jan 14 13:10:37.774418 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on false in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 1103
[Sun Jan 14 13:10:37.774531 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 506
[Sun Jan 14 13:10:37.774590 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 521
[Sun Jan 14 13:10:37.774642 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 539
[Sun Jan 14 13:10:37.774698 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 554
[Sun Jan 14 13:10:37.774762 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on false in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 798
[Sun Jan 14 13:10:37.774816 2024] [php:warn] [pid 511968] [client 54.36.148.124:44223] PHP Warning:  Trying to access array offset on false in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 1103
[Sun Jan 14 13:12:50.905457 2024] [php:warn] [pid 521020] [client 212.79.110.127:34280] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/d0/de/3f/d0de3f3fd98d576056fb02f629b93c7eb0442beb_2.file.threaded_conversation.tpl.php on line 46, referer: https://schmaker.eu/network
[Sun Jan 14 13:12:50.910537 2024] [php:warn] [pid 521020] [client 212.79.110.127:34280] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/d0/de/3f/d0de3f3fd98d576056fb02f629b93c7eb0442beb_2.file.threaded_conversation.tpl.php on line 46, referer: https://schmaker.eu/network
[Sun Jan 14 13:12:50.912046 2024] [php:warn] [pid 521020] [client 212.79.110.127:34280] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/d0/de/3f/d0de3f3fd98d576056fb02f629b93c7eb0442beb_2.file.threaded_conversation.tpl.php on line 46, referer: https://schmaker.eu/network
[Sun Jan 14 13:12:50.913249 2024] [php:warn] [pid 521020] [client 212.79.110.127:34280] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/d0/de/3f/d0de3f3fd98d576056fb02f629b93c7eb0442beb_2.file.threaded_conversation.tpl.php on line 46, referer: https://schmaker.eu/network
[Sun Jan 14 13:12:50.913974 2024] [php:warn] [pid 521020] [client 212.79.110.127:34280] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/d0/de/3f/d0de3f3fd98d576056fb02f629b93c7eb0442beb_2.file.threaded_conversation.tpl.php on line 46, referer: https://schmaker.eu/network
[Sun Jan 14 13:12:50.914188 2024] [php:warn] [pid 521020] [client 212.79.110.127:34280] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/d0/de/3f/d0de3f3fd98d576056fb02f629b93c7eb0442beb_2.file.threaded_conversation.tpl.php on line 46, referer: https://schmaker.eu/network
[Sun Jan 14 13:12:50.914399 2024] [php:warn] [pid 521020] [client 212.79.110.127:34280] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/d0/de/3f/d0de3f3fd98d576056fb02f629b93c7eb0442beb_2.file.threaded_conversation.tpl.php on line 46, referer: https://schmaker.eu/network
[Sun Jan 14 13:12:50.915173 2024] [php:warn] [pid 521020] [client 212.79.110.127:34280] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/d0/de/3f/d0de3f3fd98d576056fb02f629b93c7eb0442beb_2.file.threaded_conversation.tpl.php on line 46, referer: https://schmaker.eu/network
[Sun Jan 14 13:12:50.915418 2024] [php:warn] [pid 521020] [client 212.79.110.127:34280] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/d0/de/3f/d0de3f3fd98d576056fb02f629b93c7eb0442beb_2.file.threaded_conversation.tpl.php on line 46, referer: https://schmaker.eu/network
[Sun Jan 14 13:12:50.915886 2024] [php:warn] [pid 521020] [client 212.79.110.127:34280] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/d0/de/3f/d0de3f3fd98d576056fb02f629b93c7eb0442beb_2.file.threaded_conversation.tpl.php on line 46, referer: https://schmaker.eu/network
[Sun Jan 14 13:13:05.621887 2024] [php:warn] [pid 524293] [client 54.36.148.234:24797] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/d0/de/3f/d0de3f3fd98d576056fb02f629b93c7eb0442beb_2.file.threaded_conversation.tpl.php on line 46
[Sun Jan 14 13:13:05.625511 2024] [php:warn] [pid 524293] [client 54.36.148.234:24797] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 506
[Sun Jan 14 13:13:05.625602 2024] [php:warn] [pid 524293] [client 54.36.148.234:24797] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 521
[Sun Jan 14 13:13:05.625659 2024] [php:warn] [pid 524293] [client 54.36.148.234:24797] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 539
[Sun Jan 14 13:13:05.625710 2024] [php:warn] [pid 524293] [client 54.36.148.234:24797] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 554
[Sun Jan 14 13:13:05.625798 2024] [php:warn] [pid 524293] [client 54.36.148.234:24797] PHP Warning:  Trying to access array offset on false in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 798
[Sun Jan 14 13:13:05.625882 2024] [php:warn] [pid 524293] [client 54.36.148.234:24797] PHP Warning:  Trying to access array offset on false in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 1103
[Sun Jan 14 13:13:05.626226 2024] [php:warn] [pid 524293] [client 54.36.148.234:24797] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 506
[Sun Jan 14 13:13:05.626284 2024] [php:warn] [pid 524293] [client 54.36.148.234:24797] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 521
[Sun Jan 14 13:13:05.626352 2024] [php:warn] [pid 524293] [client 54.36.148.234:24797] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 539
[Sun Jan 14 13:13:05.626403 2024] [php:warn] [pid 524293] [client 54.36.148.234:24797] PHP Warning:  Trying to access array offset on null in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 554
[Sun Jan 14 13:13:05.626497 2024] [php:warn] [pid 524293] [client 54.36.148.234:24797] PHP Warning:  Trying to access array offset on false in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 798
[Sun Jan 14 13:13:05.626575 2024] [php:warn] [pid 524293] [client 54.36.148.234:24797] PHP Warning:  Trying to access array offset on false in /srv/http/schmaker.eu/view/smarty3/compiled/6b/a2/b2/6ba2b21ba8bb187d1debfb1cf902b2aee0e56569_2.file.wall_thread.tpl.php on line 1103
roland,

@schmaker PR 13836 is out: github.com/friendica/friendica…

roland,

@schmaker Oh, I was to lazy, I have to add another change.

schmaker,

@roland If having problem fixed within one day means being lazy, then I'm totally fine with that 😀

Thanks, man - seriously

schmaker,

@roland So, after the worker managed to get over all of the tasks without bigger issues the issue showed again and my worker is going nuts again.

Can't say for sure, but can't the worker be trying to process deffered tasks without specified limit?

schmaker,

@roland Tried to reduce worker count to 1 and tasks per worker to 1 as well, no change.

Getting roughly 2,5 GB memory usage for the problematic worker before getting killed. Maybe it's expected behavior and I'm keeping you busy for no reason?

It definitely seems wierd to me. What's even more wierd is, that everything worked for roughly 2 days with like 6 OOM kills daily (exactly what I got for last 2 years or so 😀 )

roland,

@schmaker Have you updated your instance to latest develop code? There is a new configuration entry performance.max_response_data_size which is being set at default 1000000. You can override it in your local configuration file.

schmaker,

@roland
I didn't, I only have changed the hard-coded value

schmaker,

@roland
Just to be clear - I'm running latest release, not git develop branch

schmaker,

@roland Either server deffered the task or managed to get it through - last OOM kill was 3 hours ago

schmaker,

@roland So, just to close this case somehow - it seems like problem disappeared for now. Heaviest workers are sitting with max. 1-1,5GB RAM - I consider that normal when requests are limited to 1GB.

If it shows up again, I'll open bug report as that may be something harder to troubleshoot.

Once again, thanks guys

schmaker,

Might be related? Someone is flooding with fake servers?

roland,

@schmaker I'm not sure if that is related to your OOM messages. I guess more some few files are close to that limit which causes the OOM. So I guess we need a new configuration option here which is trivial to add.

schmaker,

Thanks for all the support so far, you guys are great.

Yet, I'm stuck at the moment and can't find the way out. Is there maybe a way to pin-point problematic worker task to shut it down? I get like hundred of PHP oom kills daily.

Or maybe it's time to actually report it as bug?

roland,

@schmaker I found a LimitStream class in , that is the library Friendica uses. We can change over that class and set a limit on read bytes (configurable!). The default should be the default for Friendica, too.

roland,

@schmaker In Friendica's code I found Network/HTTPClient/Factory/HttpClient. There is a line saying $resolver->setMaxResponseDataSize(1000000); which seem to set maximum size to 1 billion Bytes (not 1 GB!).

schmaker,

@roland Is it safe to change the line just for testing purpose to find out if it actually resolves the issue?

roland,

@schmaker Yes, you can touch that line for testing purpose.

schmaker,

@roland Thank you, I'll test that out

schmaker,

@roland It seems like setting $resolver->setMaxResponseDataSize(100000); did the trick.

I'll let it set like this through the night and we will see if there will be any more OOM-killer e-mails from my VPS provider.

Thanks for the help Roland

schmaker,

@roland Did some process watching and it seems like none of workers reached more than 20% of memory (roughly 800 MB)

roland,

@schmaker I think as a first step that static value needs to be configurable and then next maybe the code needs to be rewritten to a more flexible class, like GuzzleHTTP offers with its LimitStream class.

roland,

@schmaker So Friendica uses mattwrightURLResolver wrap around cURL and not GuzzleHTTP's client and stream to download files.

roland,

@schmaker If I'm mistaken, that class doesn't loop over read data blocks (e.g. to abort on maximum size reached). It seem to download the whole URL and then check header + body if they are bigger than maximum allowed? Or is somewhere a loop here which does that check?

See it's private function fetchURL() for instance. There is only a closure and nothing else, so far I also checked some method invoking this one and also there is no "limiting" loop.

roland,

@schmaker But I guess it won't solve the problem, see my other findings in the code. And as I said: I might be wrong, maybe I oversaw a "limiting" loop.

schmaker,

@roland That would mean my worker would not be going nuts consuming 3+gigs of RAM as it shuts itself down after configured limit, if I understand it correctly?

I still don't get it why worker bot does not respect php.ini limit in this single process, thats what puzzles me most

roland,

@schmaker Yes, that is puzzling me the most, too.

hoergen,

Or decrease the number of workers from Friendica or the workers from Apache or fpm

schmaker,

@hoergen As i noted in comment above - it seems like single process consumes all the memory even though PHP limit is set to 256MB

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • php
  • Durango
  • DreamBathrooms
  • thenastyranch
  • magazineikmin
  • osvaldo12
  • khanakhh
  • Youngstown
  • mdbf
  • slotface
  • rosin
  • everett
  • ngwrru68w68
  • kavyap
  • InstantRegret
  • JUstTest
  • GTA5RPClips
  • cubers
  • ethstaker
  • normalnudes
  • tacticalgear
  • cisconetworking
  • tester
  • Leos
  • modclub
  • megavids
  • provamag3
  • anitta
  • lostlight
  • All magazines
    •