Montée en charge en production

🚧 Installation / Administration
1

Bonjour,
Quels sont les différents leviers à mettre en place sur Dokos et/ou son environnement pour une montée en charge en production ?
En effet, sur un site, nous avons quelques soucis de rĂ©ponse avec des erreurs de type « Server interne introuvable Â», quand plusieurs personnes se connectent et agissent en mĂŞme temps sur Dokos.

1 « J'aime »
2

Bonjour @Ludovic,

Il serait intéressant de comprendre d’où vient l’erreur.
Je ne crois pas avoir déjà rencontré « Serveur interne introuvable », mais ça ressemble à une erreur courante qui est « Internal Server Error » qui indique une erreur de connexion à la base de donnée.

Vous pouvez ouvrir les logs, qui sont dans le dossier logs de votre bench.
Pour ce type d’erreur, il y en a trois qui peuvent être intéressants:

  • frappe.log
  • web.log
  • web.error.log

Ensuite, pour permettre la montée en charge du serveur, vous pouvez commencer par ajuster le nombre de workers gunicorn en modifiant la valeur de gunicorn_workers dans le fichier common_site_config.json (dossier sites)
Il est recommandé d’avoir 2*cores+1 worker par bench: Design — Gunicorn 21.2.0 documentation

Une fois la valeur modifiée, il faut regénérer la configuration de supervisor:

  • bench setup supervisor
  • sudo supervisorctl reread
  • sudo supervisorctl update

Ceci étant, il suffit d’avoir 1 ou 2 workers pour permettre un certain nombre de connexions simultanées, donc il faudrait vraiment commencer par comprendre ce qui pose problème.

Bonne journée !

1 « J'aime »
3

Bonjour Charles-Henri,
Merci pour cette réponse. L’erreur est bien « Internal Server Error ». Je pense que c’est effectivement lié à des erreurs, car la valeur gunicorn_workers est bien paramétrée.

Ci-dessous l’ensemble des logs sur une seule requête « /app » :

[2023-09-11 10:17:08 +0200] [716030] [ERROR] Error handling request /api/method/frappe.desk.form.load.getdoctype?doctype=Event&with_parent=1&cached_timestamp=&_=1694420227983
Traceback (most recent call last):
  File "/home/erp/frappe-bench/apps/frappe/frappe/app.py", line 66, in application
    response = frappe.api.handle()
  File "/home/erp/frappe-bench/apps/frappe/frappe/api.py", line 53, in handle
    return _RESTAPIHandler(call, doctype, name).get_response()
  File "/home/erp/frappe-bench/apps/frappe/frappe/api.py", line 69, in get_response
    return self.handle_method()
  File "/home/erp/frappe-bench/apps/frappe/frappe/api.py", line 79, in handle_method
    return frappe.handler.handle()
  File "/home/erp/frappe-bench/apps/frappe/frappe/handler.py", line 47, in handle
    data = execute_cmd(cmd)
  File "/home/erp/frappe-bench/apps/frappe/frappe/handler.py", line 85, in execute_cmd
    return frappe.call(method, **frappe.form_dict)
  File "/home/erp/frappe-bench/apps/frappe/frappe/__init__.py", line 1638, in call
    return fn(*args, **newargs)
  File "/home/erp/frappe-bench/apps/frappe/frappe/desk/form/load.py", line 75, in getdoctype
    frappe.response.docs.extend(docs)
AttributeError: 'NoneType' object has no attribute 'extend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/erp/frappe-bench/apps/frappe/frappe/utils/jinja.py", line 95, in render_template
    return get_jenv().from_string(template).render(context)
  File "/home/erp/frappe-bench/env/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/home/erp/frappe-bench/env/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/home/erp/frappe-bench/apps/ipcem/ipcem/templates/web.html", line 1, in top-level template code
    {% extends base_template_path %}
  File "/home/erp/frappe-bench/env/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/home/erp/frappe-bench/apps/frappe/frappe/templates/base.html", line 18, in template
    href="{{ (favicon or "/assets/frappe/images/favicon.png") | abs_url }}"
jinja2.exceptions.TemplateAssertionError: No filter named 'abs_url'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/erp/frappe-bench/apps/frappe/frappe/website/serve.py", line 18, in get_response
    response = renderer_instance.render()
  File "/home/erp/frappe-bench/apps/frappe/frappe/website/page_renderers/template_page.py", line 84, in render
    html = self.get_html()
  File "/home/erp/frappe-bench/apps/frappe/frappe/website/utils.py", line 544, in cache_html_decorator
    html = func(*args, **kwargs)
  File "/home/erp/frappe-bench/apps/frappe/frappe/website/page_renderers/template_page.py", line 101, in get_html
    html = self.render_template()
  File "/home/erp/frappe-bench/apps/frappe/frappe/website/page_renderers/template_page.py", line 239, in render_template
    html = frappe.render_template(self.source, self.context, safe_render=safe_render)
  File "/home/erp/frappe-bench/apps/frappe/frappe/utils/jinja.py", line 97, in render_template
    throw(
  File "/home/erp/frappe-bench/apps/frappe/frappe/__init__.py", line 517, in throw
    msgprint(
  File "/home/erp/frappe-bench/apps/frappe/frappe/__init__.py", line 485, in msgprint
    _raise_exception()
  File "/home/erp/frappe-bench/apps/frappe/frappe/__init__.py", line 432, in _raise_exception
    raise raise_exception(msg)
frappe.exceptions.ValidationError: <pre>{% extends "templates/web.html" %}

{% block title %}{{ title or _("Message") }}{% endblock %}

{% block page_content %}
<style>
{% include "templates/styles/card_style.css" %}
{% if fullpage %}
header, footer {
        display: none;
}
html, body {
        background-color: #f5f7fa;
}
{% endif %}
{% if card_width %}
.page-card {
        max-width: {{ card_width }}px;
}
{% endif %}
</style>
<div class='page-card'>
        <h4 class='page-card-head'>
                <span class='indicator {{ indicator_color or "blue" }}'>
                        {{ title or _("Message") }}</span>
        </h4>
        <div class="page-card-body">
        {% block message_body %}
                <p>{{ message or "" }}</p>
                {% if primary_action %}
                <div><a href='{{ primary_action or "/" }}' class='btn btn-primary btn-sm btn-block'>
                        {{ primary_label or _("Home") }}</a></div>
                {% endif %}
        {% endblock %}
        </div>
</div>
{% if error_code %}
<p class='text-muted text-center small' style='margin-top: -20px;'>{{ _("Status: {0}").format(error_code) }}</p>
{% endif %}
<script>
        frappe.ready(function() {
                if (window.location.hash || window.location.href.includes('/app')) {
                        localStorage.setItem('session_last_route', window.location.pathname + window.location.hash + window.location.search);
                }

                $('.btn-primary').focus();
        });
</script>
{% endblock %}
</pre><pre>Traceback (most recent call last):
  File "apps/frappe/frappe/app.py", line 66, in application
    response = frappe.api.handle()
  File "apps/frappe/frappe/api.py", line 53, in handle
    return _RESTAPIHandler(call, doctype, name).get_response()
  File "apps/frappe/frappe/api.py", line 69, in get_response
    return self.handle_method()
  File "apps/frappe/frappe/api.py", line 79, in handle_method
    return frappe.handler.handle()
  File "apps/frappe/frappe/handler.py", line 47, in handle
    data = execute_cmd(cmd)
  File "apps/frappe/frappe/handler.py", line 85, in execute_cmd
    return frappe.call(method, **frappe.form_dict)
  File "apps/frappe/frappe/__init__.py", line 1638, in call
    return fn(*args, **newargs)
  File "apps/frappe/frappe/desk/form/load.py", line 75, in getdoctype
    frappe.response.docs.extend(docs)
AttributeError: 'NoneType' object has no attribute 'extend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apps/frappe/frappe/utils/jinja.py", line 95, in render_template
    return get_jenv().from_string(template).render(context)
  File "env/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "env/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "apps/ipcem/ipcem/templates/web.html", line 1, in top-level template code
    {% extends base_template_path %}
  File "env/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "apps/frappe/frappe/templates/base.html", line 18, in template
    href="{{ (favicon or "/assets/frappe/images/favicon.png") | abs_url }}"
jinja2.exceptions.TemplateAssertionError: No filter named 'abs_url'.
</pre>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/erp/frappe-bench/apps/frappe/frappe/utils/jinja.py", line 95, in render_template
    return get_jenv().from_string(template).render(context)
  File "/home/erp/frappe-bench/env/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/home/erp/frappe-bench/env/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/home/erp/frappe-bench/apps/ipcem/ipcem/templates/web.html", line 1, in top-level template code
    {% extends base_template_path %}
  File "/home/erp/frappe-bench/env/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/home/erp/frappe-bench/apps/frappe/frappe/templates/base.html", line 18, in template
    href="{{ (favicon or "/assets/frappe/images/favicon.png") | abs_url }}"
jinja2.exceptions.TemplateAssertionError: No filter named 'abs_url'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/erp/frappe-bench/env/lib/python3.10/site-packages/gunicorn/workers/sync.py", line 136, in handle
    self.handle_request(listener, req, client, addr)
  File "/home/erp/frappe-bench/env/lib/python3.10/site-packages/gunicorn/workers/sync.py", line 179, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/home/erp/frappe-bench/env/lib/python3.10/site-packages/werkzeug/local.py", line 237, in application
    return ClosingIterator(app(environ, start_response), self.cleanup)
  File "/home/erp/frappe-bench/env/lib/python3.10/site-packages/werkzeug/wrappers/request.py", line 187, in application
    resp = f(*args[:-2] + (request,))
  File "/home/erp/frappe-bench/apps/frappe/frappe/app.py", line 84, in application
    response = handle_exception(e)
  File "/home/erp/frappe-bench/apps/frappe/frappe/app.py", line 254, in handle_exception
    response = frappe.utils.response.report_error(http_status_code)
  File "/home/erp/frappe-bench/apps/frappe/frappe/utils/response.py", line 44, in report_error
    response = build_response("json")
  File "/home/erp/frappe-bench/apps/frappe/frappe/utils/response.py", line 64, in build_response
    return response_type_map[frappe.response.get("type") or response_type]()
  File "/home/erp/frappe-bench/apps/frappe/frappe/utils/response.py", line 199, in as_page
    return get_response(
  File "/home/erp/frappe-bench/apps/frappe/frappe/website/serve.py", line 26, in get_response
    response = ErrorPage(exception=e).render()
  File "/home/erp/frappe-bench/apps/frappe/frappe/website/page_renderers/template_page.py", line 84, in render
    html = self.get_html()
  File "/home/erp/frappe-bench/apps/frappe/frappe/website/utils.py", line 544, in cache_html_decorator
    html = func(*args, **kwargs)
  File "/home/erp/frappe-bench/apps/frappe/frappe/website/page_renderers/template_page.py", line 101, in get_html
    html = self.render_template()
  File "/home/erp/frappe-bench/apps/frappe/frappe/website/page_renderers/template_page.py", line 239, in render_template
    html = frappe.render_template(self.source, self.context, safe_render=safe_render)
  File "/home/erp/frappe-bench/apps/frappe/frappe/utils/jinja.py", line 97, in render_template
    throw(
  File "/home/erp/frappe-bench/apps/frappe/frappe/__init__.py", line 517, in throw
    msgprint(
  File "/home/erp/frappe-bench/apps/frappe/frappe/__init__.py", line 485, in msgprint
    _raise_exception()
  File "/home/erp/frappe-bench/apps/frappe/frappe/__init__.py", line 432, in _raise_exception
    raise raise_exception(msg)
frappe.exceptions.ValidationError: <pre>{% extends "templates/web.html" %}

{% block title %}{{ _("Error") }}{% endblock %}

{%- block head_include %}
<link rel="stylesheet" href="/assets/frappe/css/hljs-night-owl.css">
{% endblock -%}

{% block page_content %}
<style>
        .error-content {
                border-radius: 8px;
                background-color: #f5f7fa;
                margin: 3rem auto;
        }

        code::-webkit-scrollbar {
                display: none;
        }
        {% include "templates/styles/card_style.css"%}
</style>
<script></script>
<div class="page-card">
        <div class="page-card-head">
                <span class="indicator red">{{ error_title }}</span>
        </div>
        <p>{{ error_message }}</p>
        <div>
                <a href="/" class="btn btn-primary btn-sm">{{ _("Home") }}</a>
        </div>
</div>
<p class="text-muted text-center small" style="margin-top: -20px;">
        {{ _("Error Code: {0}").format(http_status_code) }}
</p>
<div class="text-center mt-3">
        <button class="btn btn-xs btn-link text-muted small view-error" >{{ _("Show Traceback") if not dev_server else _("Hide Traceback") }}</a>
</div>
<div class="error-content {{ 'hidden' if not dev_server else '' }}">
        <pre><code>{{ error }}</code></pre>
</div>
{% endblock %}

{% block script %}
<script>
        let toggle_button = $(".view-error");
        let error_log = $(".error-content");

        toggle_button.on('click', () => {
                if (error_log.hasClass("hidden")) {
                        toggle_button.html(`{{ _("Hide Traceback") }}`);
                        error_log.removeClass("hidden");
                } else {
                        toggle_button.html(`{{ _("Show Traceback") }}`);
                        error_log.addClass("hidden");
                }
        })
</script>
{% endblock %}
</pre><pre>Traceback (most recent call last):
  File "apps/frappe/frappe/app.py", line 66, in application
    response = frappe.api.handle()
  File "apps/frappe/frappe/api.py", line 53, in handle
    return _RESTAPIHandler(call, doctype, name).get_response()
  File "apps/frappe/frappe/api.py", line 69, in get_response
    return self.handle_method()
  File "apps/frappe/frappe/api.py", line 79, in handle_method
    return frappe.handler.handle()
  File "apps/frappe/frappe/handler.py", line 47, in handle
    data = execute_cmd(cmd)
  File "apps/frappe/frappe/handler.py", line 85, in execute_cmd
    return frappe.call(method, **frappe.form_dict)
  File "apps/frappe/frappe/__init__.py", line 1638, in call
    return fn(*args, **newargs)
  File "apps/frappe/frappe/desk/form/load.py", line 75, in getdoctype
    frappe.response.docs.extend(docs)
AttributeError: 'NoneType' object has no attribute 'extend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apps/frappe/frappe/utils/jinja.py", line 95, in render_template
    return get_jenv().from_string(template).render(context)
  File "env/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "env/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "apps/ipcem/ipcem/templates/web.html", line 1, in top-level template code
    {% extends base_template_path %}
  File "env/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "apps/frappe/frappe/templates/base.html", line 18, in template
    href="{{ (favicon or "/assets/frappe/images/favicon.png") | abs_url }}"
jinja2.exceptions.TemplateAssertionError: No filter named 'abs_url'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apps/frappe/frappe/website/serve.py", line 18, in get_response
    response = renderer_instance.render()
  File "apps/frappe/frappe/website/page_renderers/template_page.py", line 84, in render
    html = self.get_html()
  File "apps/frappe/frappe/website/utils.py", line 544, in cache_html_decorator
    html = func(*args, **kwargs)
  File "apps/frappe/frappe/website/page_renderers/template_page.py", line 101, in get_html
    html = self.render_template()
  File "apps/frappe/frappe/website/page_renderers/template_page.py", line 239, in render_template
    html = frappe.render_template(self.source, self.context, safe_render=safe_render)
  File "apps/frappe/frappe/utils/jinja.py", line 97, in render_template
    throw(
  File "apps/frappe/frappe/__init__.py", line 517, in throw
    msgprint(
  File "apps/frappe/frappe/__init__.py", line 485, in msgprint
    _raise_exception()
  File "apps/frappe/frappe/__init__.py", line 432, in _raise_exception
    raise raise_exception(msg)
frappe.exceptions.ValidationError: <pre>{% extends "templates/web.html" %}

{% block title %}{{ title or _("Message") }}{% endblock %}

{% block page_content %}
<style>
{% include "templates/styles/card_style.css" %}
{% if fullpage %}
header, footer {
        display: none;
}
html, body {
        background-color: #f5f7fa;
}
{% endif %}
{% if card_width %}
.page-card {
        max-width: {{ card_width }}px;
}
{% endif %}
</style>
<div class='page-card'>
        <h4 class='page-card-head'>
                <span class='indicator {{ indicator_color or "blue" }}'>
                        {{ title or _("Message") }}</span>
        </h4>
        <div class="page-card-body">
        {% block message_body %}
                <p>{{ message or "" }}</p>
                {% if primary_action %}
                <div><a href='{{ primary_action or "/" }}' class='btn btn-primary btn-sm btn-block'>
                        {{ primary_label or _("Home") }}</a></div>
                {% endif %}
        {% endblock %}
        </div>
</div>
{% if error_code %}
<p class='text-muted text-center small' style='margin-top: -20px;'>{{ _("Status: {0}").format(error_code) }}</p>
{% endif %}
<script>
        frappe.ready(function() {
                if (window.location.hash || window.location.href.includes('/app')) {
                        localStorage.setItem('session_last_route', window.location.pathname + window.location.hash + window.location.search);
                }

                $('.btn-primary').focus();
        });
</script>
{% endblock %}
</pre><pre>Traceback (most recent call last):
  File "apps/frappe/frappe/app.py", line 66, in application
    response = frappe.api.handle()
  File "apps/frappe/frappe/api.py", line 53, in handle
    return _RESTAPIHandler(call, doctype, name).get_response()
  File "apps/frappe/frappe/api.py", line 69, in get_response
    return self.handle_method()
  File "apps/frappe/frappe/api.py", line 79, in handle_method
    return frappe.handler.handle()
  File "apps/frappe/frappe/handler.py", line 47, in handle
    data = execute_cmd(cmd)
  File "apps/frappe/frappe/handler.py", line 85, in execute_cmd
    return frappe.call(method, **frappe.form_dict)
  File "apps/frappe/frappe/__init__.py", line 1638, in call
    return fn(*args, **newargs)
  File "apps/frappe/frappe/desk/form/load.py", line 75, in getdoctype
    frappe.response.docs.extend(docs)
AttributeError: 'NoneType' object has no attribute 'extend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apps/frappe/frappe/utils/jinja.py", line 95, in render_template
    return get_jenv().from_string(template).render(context)
  File "env/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "env/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "apps/ipcem/ipcem/templates/web.html", line 1, in top-level template code
    {% extends base_template_path %}
  File "env/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "apps/frappe/frappe/templates/base.html", line 18, in template
    href="{{ (favicon or "/assets/frappe/images/favicon.png") | abs_url }}"
jinja2.exceptions.TemplateAssertionError: No filter named 'abs_url'.
</pre>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apps/frappe/frappe/utils/jinja.py", line 95, in render_template
    return get_jenv().from_string(template).render(context)
  File "env/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "env/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "apps/ipcem/ipcem/templates/web.html", line 1, in top-level template code
    {% extends base_template_path %}
  File "env/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "apps/frappe/frappe/templates/base.html", line 18, in template
    href="{{ (favicon or "/assets/frappe/images/favicon.png") | abs_url }}"
jinja2.exceptions.TemplateAssertionError: No filter named 'abs_url'.
</pre>

Il y a de fortes chances que cela provienne de la migration ERPNext vers Dokos…

$ bench version
erpnext 3.36.2
frappe 3.38.1
payments 1.2.1
bench 2.5.3
Python 3.10.8
4

Re-bonjour @Ludovic,

Dans le log d’erreur, l’erreur semble, notamment, toucher une application personnalisée:
image

J’ai l’impression que vous avez personnalisé le template « web », peut-être que la source du problème est à chercher de ce côté là ?
Si vous voulez, envoyez moi un message Ă  chdecultot@dokos.io et on peut regarder ensemble rapidement ce qui coince.

1 « J'aime »
5

Bonjour,

Après investigation et recherche, l’erreur « Internal Server Error » s’agissait bien d’un soucis dans le code de l’application personnalisée (une méthode dans l’appli perso qui tourne en boucle sur des évènements). Inutile que je publie la solution, ça ne servirait à personne :rofl:

Pour le gabarit web.html, il a été effectivement personnalisé. Je n’ai plus l’erreur depuis la résolution de la première erreur. Si le problème se reproduit et si je n’arrive pas à trouver la solution, je me permettrais de vous envoyer un message.

Merci :slight_smile:
Ludovic

1 « J'aime »