eval-dev-quality icon indicating copy to clipboard operation
eval-dev-quality copied to clipboard

Deal with dependencies requested by LLMs

Open ahumenberger opened this issue 1 year ago • 2 comments

Assume an LLM responds with something like

	package com.eval;

	import org.junit.jupiter.api.Test;
	import org.mockito.InMockito;

	import static org.junit.jupiter.api.Assertions.*;

	class PlainTest {

	    @Test
	    void testPlain() {
	        InMockito.when(Plain::plain).thenReturn(null);
	        Plain.plain();
	        InMockito.verify(Plain::plain);
	    }
	}

The test file needs mockito to run, and we need to make sure that this is available when executing the tests.

Response from custom-fireworks_accounts_fireworks_models_qwen2-72b-instruct

ahumenberger avatar Jun 10 '24 12:06 ahumenberger

What's also bad is that we just go mod tidy all dependencies for Go... so LLMs can use whatever they want, but for Java everything except Java is forbidden.

bauersimon avatar Jun 10 '24 12:06 bauersimon

llama-3-70b-instruct responded with tests using hamcrest

	import org.junit.jupiter.api.Test;
	import static org.junit.jupiter.api.Assertions.*;
	import static org.hamcrest.CoreMatchers.is;
	import static org.hamcrest.CoreMatchers.equalTo;
	import static org.hamcrest.MatcherAssert.assertThat;

ahumenberger avatar Jun 11 '24 07:06 ahumenberger