DefectDojo
Generated dojo_tagulous_test_tags slug too long
Be informative When adding test tags while reimporting and the generated tag slug already exists defectdojo starts to appends a count to the dojo_tagulous_test_tags.slug field (_nnnn) beginning with 1 to 9999. We've reached 10000 now and get value too long errors when reimporting.
Bug description
When truncating the generated slug and append a number larger than four digits the string is 51 characters long. the slug field is character varying(50).
Steps to reproduce Steps to reproduce the behavior:
- create more than 10k tests in a loop with identical tags
- See error logs of your database or defectdojo-django uwsgi containers
Expected behavior/ possible fix Check the slug size before importing. Truncate one more character when reaching this limit or increase the slug fields max length.
We're running defectdojo version 2.37.1.
Logs
uwsgi:
[28/Aug/2024 06:32:29] ERROR [dojo.api_v2.exception_handler:43] value too long for type character varying(50)
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 916, in get_or_create
return self.get(**kwargs), False
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 637, in get
raise self.model.DoesNotExist(
dojo.models.Tagulous_Test_tags.DoesNotExist: Tagulous_Test_tags matching query does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/psycopg/cursor.py", line 97, in execute
raise ex.with_traceback(None)
psycopg.errors.StringDataRightTruncation: value too long for type character varying(50)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/rest_framework/mixins.py", line 19, in create
self.perform_create(serializer)
File "/app/dojo/api_v2/views.py", line 2802, in perform_create
serializer.save(push_to_jira=push_to_jira)
File "/app/dojo/api_v2/serializers.py", line 2605, in save
self.process_scan(auto_create_manager, data, context)
File "/app/dojo/api_v2/serializers.py", line 2568, in process_scan
).process_scan(
^^^^^^^^^^^^^
File "/app/dojo/importers/default_importer.py", line 106, in process_scan
self.parsed_findings = self.parse_findings(scan, parser)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/dojo/importers/default_importer.py", line 311, in parse_findings
self.parsed_findings = self.parse_findings_static_test_type(scan, parser)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/dojo/importers/default_importer.py", line 325, in parse_findings_static_test_type
self.test = self.create_test(self.scan_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/dojo/importers/default_importer.py", line 63, in create_test
self.test = Test.objects.create(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/tagulous/models/tagged.py", line 228, in create
getattr(obj, field_name).save()
File "/usr/local/lib/python3.11/site-packages/tagulous/models/managers.py", line 538, in save
new_tags = self._ensure_tags_in_db(self.tags)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/tagulous/models/managers.py", line 569, in _ensure_tags_in_db
db_tag, __ = self.tag_model.objects.get_or_create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 923, in get_or_create
return self.create(**params), True
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 658, in create
obj.save(force_insert=True, using=self.db)
File "/usr/local/lib/python3.11/site-packages/tagulous/models/models.py", line 447, in save
return super(BaseTagModel, self).save(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save
self.save_base(
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 877, in save_base
updated = self._save_table(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1020, in _save_table
results = self._do_insert(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1061, in _do_insert
return manager._insert(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1805, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
with self.db.wrap_database_errors:
File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/psycopg/cursor.py", line 97, in execute
raise ex.with_traceback(None)
django.db.utils.DataError: value too long for type character varying(50)
[28/Aug/2024 06:32:29] ERROR [django.request:241] Internal Server Error: /api/v2/reimport-scan/
[pid: 19|app: -|req: -/-] 10.1.2.3 {64 vars in 1393 bytes} [Wed Aug 28 06:32:29 2024] POST /api/v2/reimport-scan/ => generated 59 bytes in 286 msecs (HTTP/1.1 500) 8 headers in 253 bytes (1 switches on core 1)
postgres:
2024-08-28 05:17:33 UTC:10.9.111.73(12487):admin@dd:[10584]:ERROR: value too long for type character varying(50)
2024-08-28 05:17:33 UTC:10.9.111.73(12487):admin@dd:[10584]:STATEMENT: INSERT INTO "dojo_tagulous_test_tags" ("name", "slug", "count", "protected") VALUES ('image_name=dockerregistry.xxxxxxxxxxxxxxxxxxxx.yz_5000/xxxxxxxxxxxx/xxx-service-xxxxxxxxxxxxxx', 'image_namedockerregistryxxxxxxxxxxxxxxxxxxxx__10000', 0, false) RETURNING "dojo_tagulous_test_tags"."id"
since django-tagulous v1.3.3 there is no limit for the appended number on non-unique slugs. in the settings.py it looks like this is configurable, e.g. truncate n more characters. but i dont really know about the impact.
HI @gietschess I have attempted to reproduce on 2.38.1 and was unable to unable to experience the same error. I have completed two tests:
- generated 11,000 tests with the same tag that is 49 characters long:
- generated 11,000 tests with the same tag that is 58 characters long:
Neither test generated any errors, and it also appears that the slug was truncated properly as well:
Please let me know if there is something I am missing in my testing
I was talking to @mtesauro about this one and it occurred to me that the version of tagulous on the dev branch (where I was testing) is on version 2.1.0, but version 1.3.3 us used in master/bugfix branches.
This leads me to believe that this issue will be resolved in the next minor release in October.
Hi @Maffooch ,
we've temporarly added a local_settings.py containing TAGULOUS_SLUG_TRUNCATE_UNIQUE=7 in our container to overwrite the setting as we weren't able to reimport some findings anymore.
I'll test this again with the previous setting (TAGULOUS_SLUG_TRUNCATE_UNIQUE=5) when 2.39.0 was released.
EDIT: a information on your test: when the tag name is completly identical no new slug will be generated. if you'd change to a 60 character tag name and add a hash at the end the error would occur. so the first 50 characters must be the same but there must be a difference in the last characters so that the slug would be the same when the full tag name differs.