ABSTRACT Malicious software typically resides stealthily on a user’s computer and interacts with the user’s computing resources. Our goal in this work is to improve the trustworthiness of a host and its system data. Specifically, we provide a new mechanism that ensures the correct origin or provenance of critical system information and prevents adversaries from utilizing host resources. We define data-provenance integrity as the security property stating that the source where a piece of data is generated cannot be spoofed or tampered with. We describe a cryptographic provenance verification approach for ensuring system properties and system-data integrity at kernel level. Its two concrete applications are demonstrated in the keystroke integrity verification and malicious traffic detection. EXISTING SYSTEM Existing root kit detection work includes identifying suspicious system call execution patterns, discovering vulnerable kernel hooks, exploring kernel in variants, or using a virtual machine to enforce correct system behaviors. In existing some time suspicious data not detected. PROPOSED SYSTEM In proposed system using key exchanging, cryptography, and signature technique are used. So easily detect the suspicious data. In verify module detect the suspicious data and provenance data. Receiving packet data suspicious data means placed in suspicious box. Suppose data will be provenance data means placed in provenance box. MODULE DESCRIPTION: Cryptography Cryptography is the practice and study of techniques for secure communication in the presence of third parties More generally, it is about constructing and analyzing protocols that overcome the influence of adversaries and which are related to various aspects in information security such as data confidentiality, data integrity, authentication, and non-repudiation. Modern cryptography intersects the disciplines of mathematics, computer science, and electrical engineering. Applications of cryptography include ATM cards, computer passwords, and electronic commerce. Key generation RSA involves a public key and a private key. The public key can be known to everyone and is used for encrypting messages. Messages encrypted with the public key can only be decrypted using the private key. The keys for the RSA algorithm are generated the following way: 1. Choose two distinct prime numbers p and q. o For security purposes, the integers p and q should be chosen at random, and should be of similar bit-length. Prime integers can be efficiently found using a primality test. 2. Compute n = pq. o n is used as the modulus for both the public and private keys 3. Compute φ(n) = (p – 1)(q – 1), where φ is Euler's totient function. 4. Choose an integer e such that 1 < e < φ(n) and greatest common divisor of (e, φ(n)) = 1; i.e., e and φ(n) are coprime. o e is released as the public key exponent. o e having a short bit-length and small Hamming weight results in more efficient encryption - most commonly 0x10001 = 65,537. However, small values of e (such as 3) have been shown to be less secure in some settings.[4] 5. Determine d as: i.e., d is the multiplicative inverse of e mod φ(n). This is more clearly stated as solve for d given (de) = 1 mod φ(n) This is often computed using the extended Euclidean algorithm. d is kept as the private key exponent. By construction, d*e= 1 mod φ(n). The public key consists of the modulus n and the public (or encryption) exponent e. The private key consists of the modulus n and the private (or decryption) exponent d which must be kept secret. (p, q, and φ(n) must also be kept secret because they can be used to calculate d.) An alternative, used by PKCS#1, is to choose d matching de ≡ 1 mod λ with λ = lcm(p − 1, q − 1), where lcm is the least common multiple. Using λ instead of φ(n) allows more choices for d. λ can also be defined using the Carmichael function, λ(n). The ANSI X9.31 standard prescribes, IEEE 1363 describes, and PKCS#1 allows, that p and q match additional requirements: be strong primes, and be different enough that Fermat factorization fails. Encryption Alice transmits her public key wishes to send message M to Alice. to Bob and keeps the private key secret. Bob then He first turns M into an integer m, such that by using an agreed-upon reversible protocol known as a padding scheme. He then computes the ciphertext corresponding to . This can be done quickly using the method of exponentiation by squaring. Bob then transmits to Alice. Note that at least nine values of m could yield a ciphertext c equal to m,[5] but this is very unlikely to occur in practice. Decryption Alice can recover from by using her private key exponent via computing . Given , she can recover the original message M by reversing the padding scheme. Sign Module In sign module following process are preformed. 1. Key generation, 2.encryption, 3.key exchanging 4.signature 5.send to verify module Verify Module In verify modules following process are preformed. 1. Key generation, 2.decryption, 3.key exchanging 4.send to receiver module Receiver Module In receiver module receive a packet data suspicious means place in suspicious box suppose data correct data means placed in province box. Data-provenance Setup: the data producer sets up its signing key k and data consumer sets up its verification key k0 in a secure fashion that prevents malware from accessing the secret keys. Sign(D, k): the data producer signs its data D with a secret key k, and outputs D along with its proof sig. Verify(sig, D, k0): the data consumer uses key k0 to verify the signature sig of received data D to ensure its origin, and rejects the data if the verification fails. System Configuration:H/W System Configuration:Processor - Pentium –III Speed - 1.1 Ghz RAM - 256 MB(min) Hard Disk - 20 GB Floppy Drive - 1.44 MB Key Board - Standard Windows Keyboard Mouse - Two or Three Button Mouse Monitor - SVGA S/W System Configuration:Operating System :Windows XP Front End : JAVA,RMI, SWING CONCLUSION We described a general approach for improving the assurance of system data and properties of a host, which has applications in preventing and identifying malware activities. Our host-based system security solutions against malware complement network-traffic based analysis. We demonstrated CPV’s application in identifying stealthy malware activities of a host, in particular how to distinguish malicious/unauthorized data flow from legitimate one on a computer that may be compromised. We made the following technical contributions in this paper. i) We proposed the model and operations of cryptographic provenance verification in a host-based security setting. We pointed out its important usage for achieving highly assured kernel data and application data of a host, and associated technical challenges. ii) We demonstrated our provenance verification approach in a lightweight framework for ensuring the integrity of outbound packets of a host. This traffic-monitoring framework creates checkpoints that cannot be bypassed by malware traffic. iii) We described an efficient TPM based keystroke-integrity verification protocol in a client server architecture that prevents malicious bots from forging keystroke events. This keystroke-integrity service serves as an important building block for the future construction of human-behavior driven security solutions.