[GOBBLIN-1713] Add missing sql source validation

[umustafi]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• [X] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
  • https://issues.apache.org/jira/browse/GOBBLIN-1713

Description

• [X] Here are some details about my PR, including screenshots (if applicable): Adding check to a code path to detect once a MySQL connection becomes read-only. Enforce verification when this happens, so we know to create new, replacement connections (to a read-write host). Add additional logs to warn us when this query isn't set or if an exception occurs because of a connection issue.

This calls the parent class non-default constructor to utilize this check. It will overwrite the driverClassName, url, username, and password values set by the constructor. However it will keep the other properties.

  if (!Boolean.parseBoolean(properties.getProperty(SKIP_VALIDATION_QUERY, "false"))) {
      // MySQL server can timeout a connection so need to validate connections before use
      final String validationQuery = MysqlDataSourceUtils.QUERY_CONNECTION_IS_VALID_AND_NOT_READONLY;
      LOG.info("setting `DataSource` validation query: '" + validationQuery + "'");
      this.basicDataSource.setValidationQuery(validationQuery);
      this.basicDataSource.setTestOnBorrow(true);
      this.basicDataSource.setTimeBetweenEvictionRunsMillis(Duration.ofSeconds(60).toMillis());
    }

Tests

• [X] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• [X] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[codecov-commenter]

Codecov Report

Merging #3567 (d4f400d) into master (5ae4ec7) will decrease coverage by 2.89%. The diff coverage is 60.00%.

@@             Coverage Diff              @@
##             master    #3567      +/-   ##
============================================
- Coverage     46.77%   43.88%   -2.90%     
+ Complexity    10512     2055    -8457     
============================================
  Files          2099      406    -1693     
  Lines         81988    17545   -64443     
  Branches       9132     2143    -6989     
============================================
- Hits          38349     7699   -30650     
+ Misses        40096     8989   -31107     
+ Partials       3543      857    -2686     

Impacted Files	Coverage Δ
...g/apache/gobblin/util/jdbc/DataSourceProvider.java	0.00% <0.00%> (ø)
...obblin/metastore/JobHistoryDataSourceProvider.java	62.50% <85.71%> (+18.05%)	:arrow_up:
...a/org/apache/gobblin/util/limiter/NoopLimiter.java	40.00% <0.00%> (-20.00%)	:arrow_down:
...lin/util/filesystem/FileSystemInstrumentation.java	92.85% <0.00%> (-7.15%)	:arrow_down:
...gobblin/runtime/spec_store/MysqlBaseSpecStore.java
...obblin/kafka/serialize/LiAvroDeserializerBase.java
...va/org/apache/gobblin/publisher/JdbcPublisher.java
.../service/monitoring/KafkaAvroJobStatusMonitor.java
.../extractor/extract/kafka/KafkaSimpleExtractor.java
...etention/dataset/ConfigurableCleanableDataset.java
... and 1689 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more