Manual Prompting
Group 2
13.12.2023
Prompting Technique
Select a sentence from the training set at random
Probability of having a Label = 0,5
Set of rules for unfair
clauses from the
Claudette Paper
Set of examples of
unfair clauses from
the validation set
Prompt
Template
TinyLlama-1.1BChat-v0.6
Store the result
(Fair, Unfair, or
Undecided)
Prompting with TinyLlama/TinyLlama-1.1B-Chatv0.6
messages = [
{
"role": "system",
"content": “You are an expert of Online Terms of Service.",
},
{"role": "user", "content": "A clause is unfair for the client if creates a
significant imbalance to the detriment of the customer. Is this clause fair to
the user : “ + clause_to_test}]
Parameters :
• temperature = 0,1 ; top_k = 10 ; top_p = 0,95 ; max_new_tokens = 16,32 or 64
• 500 iterations to have meaningful statistics.
First Results
.
Template = « <Clause>{Random Sentence}<Clause>{Question : Unfair} »
4
First Results
Template = « <Rule>A clause is considered unfair if it creates a significant imabalance for
.
the customer.<\Rule> <Clause>{Random Sentence}<\Clause>{Question : Unfair} »
5
First Results
Template = « <Rule>A clause is considered unfair if it creates a significant imabalance for
.
the customer. It can also be considered unfair for the following reasons : {rules}<\Rule>
<Clause>{Random Sentence}<\Clause>{Question : Unfair} »
6
First Results
Template = « <Rule>A clause is considered unfair if it creates a significant imabalance for
.
the customer. It can also be considered unfair for the following reasons : {rules}<\Rule>
<Clause>{Random Sentence}<\Clause>{Question : Fair} »
7
First Results
Template = « <Rule>A clause is considered unfair if it creates a significant imabalance for
the customer. It can also be considered unfair for the following reasons :
.
{adaptative_rule}<\Rule> <Clause>{Random Sentence}<\Clause>{Question : Unfair} »
8
First Results
Template = « <Rule>A clause is considered unfair if it creates a significant imabalance for
the customer. It can also be considered unfair for the following reasons :
.
{adaptative_rule}<\Rule>
<Clause>{Random Sentence}<\Clause>{Question : Fair} »
9
First Results
.
Template
= « <Example>Here is an example of an unfair clause {unfair_example}. Here is
the example of a Fair clause : {fair_example}<\Example> <Clause>{Random
Sentence}<\Clause>{Question : Fair} »
"content":"<EXAMPLE>Here is an example of an unfair clause : "+adaptative_clause+" Here is an
example of a fair clause : you are responsible for any activity that occurs through your account
and you agree you will not sell , transfer , license or assign your account , followers , username
, or any account rights .</EXAMPLE><CLAUSE>"+sentence+"<\CLAUSE>. <QUESTION>If you had to judge
whether if the clause is fair to the customer, among the answers Yes or No, the answer is :
</QUESTION>"
10
First Results
.
Template
= « <Example>Here is an example : {adaptative_example}<\Example> The
clause is : <Clause> {Random Sentence}<\Clause>{Question : Fair or Unfair } + « Let’s
think step by step » »
With « let’s think step by step »
Without « let’s think step by step »
11
Checking consistency
.
Template
= « <Example>Here is an example : {adaptative_example}<\Example> The
clause is : <Clause> {Fixed Sentence}<\Clause>{Question : Fair or Unfair} »
Temperature = 0,7
12
Checking consistency
.
Template
= « <Example>Here is an example : {adaptative_example}<\Example> The
clause is : <Clause> {Fixed Sentence}<\Clause>{Question : Fair or Unfair} »
Temperature = 0,2
13
N=10, tinyllama, 0.7, false, w augmented fair
<Example>Here is an example : {adaptative_example}<label>…</label><\Example>
The clause is :
<Clause><Category><unfair><unfair>{Fixed Sentence1}<fair>{Fixed Sentence2}<\Clause>
{Question : Fair or Unfair} <label>…</label>
14
N=50, tinyllama, 0.7, True, w augmented fair
<Example>Here is an example : {adaptative_example}<label>…</label><\Example>
The clause is :
<Clause><Category><unfair><unfair>{Fixed Sentence1}<fair>{Fixed Sentence2}<\Clause>
{Question : Fair or Unfair} <label>…</label>
15
N=50, tinyllama, 0.7, w no augmented fair
<Example>Here is an example : {adaptative_example}<label>…</label><\Example>
The clause is :
<Clause><Category><unfair><unfair>{Fixed Sentence1}<fair>{Fixed Sentence2}<\Clause>
{Question : Fair or Unfair} <label>…</label>
16
Overview
➢ Best results achieved when rules are present.
➢ This version TinyLlama seems prone to heavy bias according to the way
the question is phrased and the « insistance » over what is an unfair
clause in the template.
Limitation
➢ False proportion of labelled sentence and labels are not evenly
distributed.
➢ Automatic decision of Fair vs Unfair.
17
Dr. rer. nat. Erika Mustermann
München, 27. März 2015
18