security_monkey frequent null value in column "item_id" violates not-null constraint exception

I am frequently seeing these exceptions for securitygroup, elb, and alb. I have ran monkey db upgrade but still getting below exceptions. anyway, my setup is just 2-3 weeks old so it should not be required.

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.NotNullViolation: null value in column "item_id" violates not-null constraint
DETAIL:  Failing row contains (138221, 10, Unknown Access, null, f, null, null, 2020-08-17 11:07:28.174966, null, null, null, null, null, null, null, f).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/security_monkey/task_scheduler/tasks.py", line 343, in _audit_changes
    au.audit_objects()
  File "/usr/local/lib/python3.6/dist-packages/security_monkey/auditor.py", line 683, in audit_objects
    method(item)
  File "/usr/local/lib/python3.6/dist-packages/security_monkey/auditors/elb.py", line 184, in check_internet_scheme
    sub_issue_message=issue.issue, score=issue.score)
  File "/usr/local/lib/python3.6/dist-packages/security_monkey/auditor.py", line 1005, in link_to_support_item_issues
    issue = self.add_issue(link_score, issue_message, item)
  File "/usr/local/lib/python3.6/dist-packages/security_monkey/auditor.py", line 631, in add_issue
    self.override_scores = query.all()
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3233, in all
    return list(self)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/query.py", line 3388, in __iter__
    self.session._autoflush()
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 1597, in _autoflush
    util.raise_from_cause(e)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 1586, in _autoflush
    self.flush()
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 2479, in flush
    self._flush(objects)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 2617, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line 2577, in _flush
    flush_context.execute()
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
    rec.execute(self)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
    uow,
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
    insert,
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/persistence.py", line 1137, in _emit_insert_statements
    statement, params
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 982, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
    distilled_params,
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(psycopg2.errors.NotNullViolation) null value in column "item_id" violates not-null constraint
DETAIL:  Failing row contains (138221, 10, Unknown Access, null, f, null, null, 2020-08-17 11:07:28.174966, null, null, null, null, null, null, null, f).

[SQL: INSERT INTO itemaudit (score, issue, notes, action_instructions, background_info, origin, origin_summary, class_uuid, fixed, justified, justified_user_id, justification, justified_date, item_id, auditor_setting_id) VALUES (%(score)s, %(issue)s, %(notes)s, %(action_instructions)s, %(background_info)s, %(origin)s, %(origin_summary)s, %(class_uuid)s, %(fixed)s, %(justified)s, %(justified_user_id)s, %(justification)s, %(justified_date)s, %(item_id)s, %(auditor_setting_id)s) RETURNING itemaudit.id]
[parameters: {'score': 10, 'issue': 'Unknown Access', 'notes': None, 'action_instructions': None, 'background_info': None, 'origin': None, 'origin_summary': None, 'class_uuid': None, 'fixed': False, 'justified': False, 'justified_user_id': None, 'justification': None, 'justified_date': datetime.datetime(2020, 8, 17, 11, 7, 28, 174966), 'item_id': None, 'auditor_setting_id': None}]
(Background on this error at: http://sqlalche.me/e/gkpj)
ERROR:security_monkey:[X] Database error processing accounts Dev Account, cleaning up session.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.NotNullViolation: null value in column "item_id" violates not-null constraint
DETAIL:  Failing row contains (138221, 10, Unknown Access, null, f, null, null, 2020-08-17 11:07:28.174966, null, null, null, null, null, null, null, f).

Aug 17 '20 11:08 Deepak1100

Are you using the latest version from the develop branch?

Aug 17 '20 15:08 mikegrima

yes

Aug 17 '20 16:08 Deepak1100

I am using latest commit from develop branch.

Aug 17 '20 16:08 Deepak1100

Unfortunately, this one is very difficult to debug. You will need to run this in PyCharm and add a breakpoint to see what the state of the data is and why that's happening.

Aug 31 '20 16:08 mikegrima